© Copyright 2019 - All rights reserved.

The content contained within this book may not be reproduced, duplicated or transmitted without direct written permission from the author or the publisher.

Under no circumstances will any blame or legal responsibility be held against the publisher, or author, for any damages, reparation, or monetary loss due to the information contained within this book. Either directly or indirectly.

Legal Notice:

This book is copyright protected. This book is only for personal use. You cannot amend, distribute, sell, use, quote or paraphrase any part, or the content within this book, without the consent of the author or publisher.

Disclaimer Notice:

Please note the information contained within this document is for educational and entertainment purposes only. All effort has been executed to present accurate, up to date, and reliable, complete information. No warranties of any kind are declared or implied. Readers acknowledge that the author is not engaging in the rendering of legal, financial, medical or professional advice. The content within this book has been derived from various sources. Please consult a licensed professional before attempting any techniques outlined in this book.

By reading this document, the reader agrees that under no circumstances is the author responsible for any losses, direct or indirect, which are incurred as a result of the use of information contained within this document, including, but not limited to, — errors, omissions, or inaccuracies.

TABLE OF CONTENTS

Introduction

Outline:

Chapter 1: Python basics

Chapter 2: Data analysis with Python

Chapter 3: Data Visualization with Python

Chapter 4: Bonus chapter – Introduction to Machine
learning with Python

Chapter 1 : Python Basics

Python History

Installing Python

Windows:

Mac/Linux Os:

Windows:

The Anaconda navigator

Coding with Python: The rudiments

Statements, Commands, and Expressions

Comments

Python data types

Numbers

Integers

Floats

Complex numbers

Strings

String Concatenation

Variables

Lists

List indexing

String Indexing

Tuples

Dictionaries

Booleans

Comparison operators

Logical operators

Conditional statements and Loops

List Comprehension

Functions

Chapter 2 : Data Analysis with Python

NumPy

Package Installation

Manipulating arrays

Indexing and selecting arrays

Conditional selection

NumPy Array Operations

Pandas

Series

Data frames

Conditional selection

Missing Data

Group-By

Concatenate, Join and Merge

Reading and Writing data

Read

Write

Chapter 3 : Data Visualization with Python

Matplotlib

Seaborn

Pandas

Chapter 4 : Machine Learning with Python

So, what is machine learning?

Okay, machine learning is cool. How is it related
to data science?

Python and machine learning

Types of machine learning

Conclusion

Introduction

According to a report published by LinkedIn, data science is one of the fastest growing tech fields within the past 7 years. The need for companies to have a better understanding of data generated via their business has motivated a lot of interest in the field. However, there is a gap to be breached as the supply of competent data scientists is way lower than the demand. This makes data science a very in-demand skill, with generous compensation for the few that possess the relevant portfolio. On average, a data scientist makes about $109,000/year (according to glassdoor.com); this puts data scientists in the top paid ranks of the tech industry. This tends to bring up certain questions: On a scale of ‘rocket science’ to ‘quantum physics’, how complicated is data science? Well, if you - like many other people (myself included), have wondered what data science is, and why data scientists are so in-demand, then that question is not so far-fetched. On the contrary, however, data science is not that complicated.

To risk over-simplification, data science is just the application of various techniques – usually employing the fast and efficient data organization, visualization and interpretation of computer programs or software to transform raw data into information for decision-making. This type of information is useful to managers in corporate institutions for informed risk-assessments, profit optimization, fraud detection, etc. Imagine the great prospects data science offers such businesses to consistently be ahead of the competition (assuming the competition is not equally leveraging the awesomeness of data-science). Advertisements are better targeted at consumers, companies are more aware of their economic performance and possible trends or options in production, etc. These available prospects serve as excellent motivations for considering a career in data science. However, without the right tools, guidance, and dedication, mastering the skills required for data science would be a very tedious and lengthy task. This is the reason for writing this book; to bring you up to speed on the skills and tools to begin a journey into the exciting and rewarding world of data science.

At this point, you are probably quite excited about data science and the secrets it has to offer, however, you may ask “what does Python have to do with it?”

Remember, data science leverages the exceptional processing and data manipulation capacity of computers? To do this, the data scientist must communicate the task in a clear and logical manner to the computer. The problem arises in the fact that computers do not understand human language; thus, the need for programming languages that allow powerful communications between people and computers.

Python is a high-level programming language that reduces the complexity of coding using almost human expressions in its syntax. This allows programmers to build complex algorithms and systems with exceptional performances, while reducing coding time and resources. Also, Python has an extensive library of tools and applications that makes data science less technical for beginners or intermediate programmers. Therefore, most data scientists have Python as their primary tool for big data, data analysis, etc.

Now, here is the red pill – blue pill moment. This book has been carefully designed to take you on a journey to leveraging the potentials of Python for data science. However, the extent of your progress also depends on you. Each chapter is outlined to introduce a certain concept, and helpful tips would be given to allow you avoid common Noob-mistakes (Noob is a programming jargon for beginner). Also, there would be practice exercises for you to test yourself after each chapter. This is meant to serve as a motivation for practicing the techniques or lessons that were introduced in the chapter.

It is expected that, by following every section of this book and practicing all the lessons presented, as well as practice questions included, you would not only have learned the basic techniques required for using Python for data science, but you would also have confidence to build your own practical systems and projects using Python.

The outline of this book is detailed below, and it is a guide for maximizing your use of this book depending on your level in programming. On this note, I wish you Godspeed as you journey through this book to becoming a data scientist with Python.

Outline:

Chapter 1: Python basics

This chapter introduces you to the basics of Python. The scope of this chapter spans all the necessary details for getting started with Python: downloading and installation of relevant Python packages and tools i.e. Anaconda, managing Python environments, and actual coding with Python. This is especially important for those with little or no programming experience. Even with programming experience (but new to Python), it is advisable to read this chapter. Most of the lessons from this chapter would be recalled in subsequent chapters as we dive deeper into the applications of Python for data science. However, if you already know the basics of Python and can already handle practical problems with Python, then you can skip this chapter. At the end of this chapter, a basic project would be completed using Python, and you can also try out some of the exercises that follow. These exercises would incorporate all the lessons from the chapter and doing them would greatly speed up your progress.

Chapter 2: Data analysis with Python

Here, the skills and techniques learned in chapter 1 are leveraged for starting data science. The most important (and popular) frameworks for analyzing and structuring data with Python are introduced i.e. The NumPy and Pandas frameworks. A few practical examples on using these frameworks will be considered, and practice exercises are provided.

Chapter 3: Data Visualization with Python

This chapter introduces the reader to various plotting techniques in Python. There are various plot libraries for data visualization using Python, these include: Matplotlib, Seaborn, and Pandas, etc. Data visualization is a significant part of data science, as it explains the analyzed data better using graphical representations. For example, it would be much easier to view the trend of user subscription to a service via a chart, graph, or even a pictogram, as opposed to just looking at numbers in a spreadsheet or table! This is the essence of the data visualization libraries available through Python, and we would be exploring their capabilities and specific usage in this chapter. Also, examples would be presented here, and relevant practice questions would be given. At the end of this chapter, it may be wise to review all the lessons from chapter 2, and once you are confident of your competence, to attempt solving some real-life problems using the skills gained from reading these chapters. After all, the whole purpose of data science is to solve real-world problems!

Chapter 4: Bonus chapter – Introduction to Machine learning with Python

This is an extra package added to this book. Machine learning is useful for data science, as it involves the use of powerful statistical techniques and tools for improving outcomes through the evaluation of big data. Through machine learning, the powerful capacity of computers for finding patterns and predicting outcomes via probability theory is leveraged. While this is not a comprehensive study on the subject, the reader is introduced to the underlying concepts, applications, and tools supported through Python for unleashing the power of artificial intelligence.

Chapter 1

Python Basics

Python History

This is a quick recap of the history of Python, along with some common terminologies/references you might find when interacting with other Python programmers.

Python was developed and introduced in the late 1980s by a Dutch programmer Guido Van Rossum . It is an object-oriented, interpreted, high-level programming language with high portability i.e. Python code/applications can be run on multiple platforms that have a version of Python installed. The name Python was also adapted from the name of a popular show “Monty Python”, which Guido was watching at the time he developed the language. So far, there have been various versions of the language, with the latest being Python 3.7 starting from 2018. While there have been debates on the best version of Python to use i.e. Python version 2xx or 3xx, there are only minor differences in either version. In this book however, we would be using the latest Python 3xx version.

Installing Python

The basic way of getting Python installed on your computer is by visiting the official website for Python www.python.org and downloading the version of Python you need. Remember to download the setup that is specific to your operating system, either Windows, Mac Os or Linux. For Linux or Mac users, you may already have a version of Python installed on your computer. After downloading and installing Python through the set-up file, the installation can be verified by opening a terminal in mac or Linux os, or the command prompt in windows. This example is Windows Os based, since it does not have Python installed by default. However, the same command works on Mac and Linux.

Windows:

Open a search by pressing Windows key + Q, search for command prompt, right-click the result and select ‘run as administrator’. The Command Prompt-Admin window opens, type:  python - -version.

If Python is properly installed, you should get a result like this: Python 3.7. x .

Mac/Linux Os:

Open a terminal by going to ‘Applications >> Utilities, then click on Terminal. Alternatively, you can open a search by pressing the Command key + Spacebar, then search for terminal and press enter. Once the terminal opens, type:  python - -version.

You are going to get a result like this: Python 2.7. x .

This version could however be the basic Python that comes with the Os. To check if Python version 3xx is installed, type: python3 - -version

You are going to get a result like this: Python 3.7. x .

To start using Python via the command prompt or terminal, type: python (without quotes). You should see something like this:

Windows:

Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23, 2018, 22:20:52) [MSC v.1916 32 bit (Intel)] on win32

Type "help", "copyright", "credits" or "license" for more information.

>>>

The three arrowheads/greater-than-signs at the bottom is the trademark of the Python console. This means you are now in a Python environment and can start writing Python codes.

Let us write our first Python code using the command prompt!

At the Python console (after typing ‘python’ at the command window), enter the following code: print (‘Hello World’) and press enter. You should get a result like this:

print (‘Hello world’)

Hello World

There, we just wrote our first Python program by displaying a message using the Python print function.

Hint:   It should be noted that, while Python 3xx uses the print function as an object, i.e. print (‘Hello World’), the print function is a statement in Python 2xx i.e. print ‘Hello World’. This is one (among a few others) of the differences between using Python 2 and Python 3. Hence, to avoid having errors in your code examples, it is important to use Python version 3xx to follow the lessons in this book.

Now, we have learned how to run Python via the command window. However, to write larger programs that we would prefer to finish developing before running (the command window runs every line of code immediately!), some text editors like notepad, notepad++ and various IDEs have been designed to create and save Python programs. The preferred extension for Python files is ‘.py’ e.g. Hello_world.py, and can be run from the console by typing the full path to the Python file in your command window. 

For example, if we saved our Hello_world program on the desktop. The file path could be something like this: C:\Users\yourcomputername\Desktop\Hello_World.py. To run our Python program:

Open the command window, and enter ‘file path’, you should see something like this:

Microsoft Windows [Version 10.0.17134.648]

(c) 2018 Microsoft Corporation. All rights reserved.

C:\Users\Username >C:\Users\Username\Desktop\Hello_World.py

Hello World

Tip: To write Python scripts with notepad: create a notepad text file and open it >> Once opened, write your Python code in the file >> at the top left corner, click file and select save as >> select the save location of your code >> In the ‘filename’ box, name the file with a ‘.py’ extension, e.g. program.py, and in the ‘Save as type’ drop-down list, select ‘All files’. Then click save.

You should see a Python file in your save location. You can also drag and drop this file into the command window and run it by pressing enter.

Exercise : Try writing fun programs to display text using the print function as described in the example.

The Anaconda navigator

This is a graphical user interface for running Python, and it is specially optimized for data science with Python. It is highly recommended for beginning data science with Python, as it makes the management of important data science packages and Python libraries easy. It is a part of the Anaconda distribution, which comes with a version of Python (the latest Anaconda still comes with Python 3.7 at this time, May,2019) i.e. if you do not have Python installed, anaconda will install a version of Python on your computer.

Figure 1: The anaconda navigator

The most significant advantage of using anaconda is the click-to-launch features it offers for running any package installed in its path. It greatly reduces the need to install packages via command prompt using the pip install method. Also, it is easy to create virtual environments for installing and running various versions of Python, and other data science programs like ‘R’. Virtual environments are paths (or folders) specific to certain applications or packages in an operating system. So, by separating various packages into their respective virtual environment, there would be no risk of conflict in running multiple packages/programs at the same time.

Anaconda also comes with an interactive IDE (Integrated development environment) for running Python codes. There is the Spyder IDE, which uses a console and script interface; the Jupyter notebook, which runs an interactive Python console and GUI in your browser; Jupyter Lab, etc. We would be using the Jupyter notebook for our subsequent Python programming in this book.

To install anaconda navigator, go to www.anaconda.com/downloads and download the latest version of anaconda distribution. After downloading, run the setup file and follow the onscreen instructions to fully install anaconda on your computer.

Once the installation is done, the next step is to check if anaconda is properly installed.

Open a search using Winkey + Q , and type in anaconda. You should see the anaconda navigator, and anaconda prompt displayed. Click on the anaconda navigator.

If the anaconda navigator opens up, then it means you have successfully installed anaconda. Alternatively, you can open the anaconda prompt to check for the version of anaconda installed, and also run anaconda navigator.

At the anaconda prompt, type: conda –version. You should get a result like this:

(base) C:\Users\Username >Conda --version

conda 4.6.14

To check the version of Python installed with the Anaconda distribution, type: python –version .

(base) C:\Users\Oguntuase>python --version

Python 3.7.0

To launch the navigator, type: anaconda-navigator

Once the navigator opens, the installation is verified and our work is done.

Now that we have installed Python via anaconda on our computer, we are ready to start our data science journey with Python. However, as stated in the outline, we would be introducing some Python fundamentals which are considered pre-requisites for the data science lessons.

Tip: While the idea for computer programming is mostly similar, the difference in programming languages exists in their syntax. Things like variable declaration, loops, etc. have valid formats in Python, and we would be exploring that soon. Ensure to follow the examples, and try out the practice exercises.

Open the Jupyter notebook either by clicking launch via the anaconda navigator, or open anaconda prompt and type: Jupyter notebook . A browser window will open as shown in figure 2 below.

Figure 2: Jupyter notebook

Once on the Jupyter notebook, at the right side click New >> then under notebook, click python 3/python[default] .

This will open a Python editor where you can write and execute codes in real-time.

Tip : Jupyter notebook files are saved as ‘. ipynb’, and can be accessed/opened by browsing to the file location via the Jupyter notebook explorer page. Jupyter also supports Markdown, and other live coding features that make it easier to annotate code. For extra information on using Jupyter, go to: https://jupyter.readthedocs.io/en/latest/running.html#running

Coding with Python: The rudiments

In this section, you can either go through the exercises using Spyder IDE, Jupyter notebook, or IDLE (for those that have Python installed outside anaconda). However, it is recommended you use the Jupyter lab to get familiar with the interface, as we would be using it for the data science section.

Statements, Commands, and Expressions

High-level programming languages are not that different from human language. They follow a particular pattern and rule. Once that rule is missed, it gives an error. In language, it is called a grammatical error; in programming, it is called a syntax error.

A statement in Python is a complete set(s) of instruction that achieves a task. It can be considered as an equivalent to the sentence in the English language that contains a Subject, verb and object.

Recall your first Python program, print (‘Hello World’)? The print () function used is a command. It can be considered as a verb, which does nothing on its own. The ‘Hello World ’ part is the expression, which is the stuff to be done . Together they make a complete instruction called a statement, which tells Python exactly what to do, and how!

There are many commands for doing various things in Python, and we would get familiar with using them as we progress. Expressions on the other hand, take on various forms. At times, it can be evaluated e.g. 4+5; other times, like the previous example, it is just some text to be displayed.

Let us try another version of the ‘Hello World’ example. How about we tell Python to display the two most dreadful words you would hope not to see on your screen while playing a game! ‘GAME OVER’

Info: Python is a great tool for game development as well. This example is a reference to that possibility in the application of Python. However, for standard game development, you would need to learn some specific Python libraries and methods. You can find extra information about this from: https://wiki.python.org/moin/PythonGames

To do this, we could just modify the expression in our former program.

In Jupyter notebook, type: print (‘GAME OVER!’) and press Shift + Enter to run. You should have a result like this:

In [1]:​ print (‘GAME OVER!’)

Out[1]:​ GAME OVER!

Tip: for the print function in Python, you can enclose your expressions in a single quote ‘’, double quote ““, or three quotes “’ “”. However, always make sure to use the same type of quotes to start and terminate the expression.

Comments

Consider this code that is run in Jupyter notebook:

In [2]:​ # This program says Hello to the world

            # Author: random_Coder_Guy

            print (‘Hello, World!’)

Out[2]:​ Hello, World!

You can easily notice that the only thing displayed in the result is the ‘Hello, World’ statement. Now let us try this:

In [3]:​ This program says Hello to the world

      ​ Author: random_Coder_Guy

      ​ print (‘Hello, World!’)

​

      File "<ipython-input-14-239b196f5fd8>", line 1

    ​ ​ This program says Hello to the world

               ​          ^

   ​ SyntaxError: invalid syntax

We get an error. Python is even kind enough to tell us our error is in the first line! The difference between these two codes is immediately obvious. The first code did not throw an error because of the pound-sign/hashtag (#) in lines 1 and 2. This is called a comment.

Comments are meant to illustrate code and improve its readability. When reading the first code, it is obvious what the program is intended to do. However, to prevent Python from executing those lines in our code (thereby leading to syntax error as in the second code), we use the comment sign to distinguish these texts.

A good question would be, are comments compulsory? The only answer is YES! Comments will save your life (while coding at least, it would not do so much in a Mexican standoff), and probably save others from tearing their hair out while trying to understand your code!

Have you ever solved a problem or played a level in a game and years later, you cannot remember how you did it? Now imagine that happening to a very important code you wrote; two years after, you cannot figure out what it does or how it works. That is a form of bad programming, and the lesson to be learned is ‘ALWAYS COMMENT YOUR CODE!’

Using the Jupyter notebook also offers more options for improving code readability. Since it uses a web-based framework, it supports ‘markdown’ (which is just a way of formatting texts on webpages). To fully explore this feature and other cool stuff with Jupyter, visit: https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/

Python data types

Data types are specifications of the kind of operations a programmer expects the computer to perform on/with specific data items. It is what determines the attributes of the elements of an expression, as declared by the programmer i.e. You.

Python, like any other programming language, is also very particular about its data types. The following are the various data types available in Python which we would be using later on:

Numbers (i.e. integers, floats, complex etc.)

Strings

Lists

Tuples

Dictionaries

Booleans

Print formatting

Numbers

Numbers are represented in Python as either integers, floats (floating-point numbers) or complex numbers.

Integers

Numbers that do not have decimal points are represented as integers in Python with the type int (). By default, Python assumes integers to be in decimal (or base 10), but the bases can be changed if there is a need to work in binary, octal or hexadecimal.

You can do arithmetic operations with integers right from the prompt (or code cell in Jupyter notebook).

# Code for arithmetic operations with Python

In [4]:​ 5 + 10

Out[4]:​ 15

In [5]:​ 20 - 12

Out[5]:​ 8

In [6]:​ 3*5

Out[6]:​ 15

In [7]:​ 20/5

Out[7]:​ 4.0

Here we can see that Python can perform basic arithmetic operations with integers. If you observe, it is clear that addition, subtraction, and multiplication of integers with another integer results in an integer. For division, however, the result is a number with a decimal point. This is a floating-point number.

Tip: There are other arithmetic operations possible with integers. All these operations are similar to those in basic mathematics. Remember Bodmas, Pedmas or Pemdas?  I like to use the acronym Bedmas for remembering the priority for Python arithmetic operations.

B rackets take first priority, followed by E xponentiation, then D ivision or M ultiplication (they have the same priority and can be interchanged), and finally A ddition and S ubtraction (which are also of same priority).

This is illustrated in the following code:

In [8]:​ # This code tries to determine order operations with integers

​ ​ (7-2*2)**3 / (12-3**2)

Out[8]:​ 9.0

Now, how does this code work? First, in-case you missed it, the ‘**’ is the sign for exponentiation in Python (some other languages allow the use of ‘^’).

Looking at the first bracket while keeping Bedmas in mind, it becomes obvious that the exponentiation inside the first bracket is treated, making it (7-4) which gives 3. Then 3 to the power of 3 gives 27.

For the second bracket, the same technique applies and results in 3. So, 27/3 gives 9 and Python returns a floating-point value of 9.0 due to the division.

Exercise: Try out a few other arithmetic operations with integers and see what you get. Remember to follow the order of arithmetic operations to avoid errors.

Initially, there was a cap to the number of characters allowable for a type int () data type, but it has been lifted in new releases of Python. Now you can declare an integer type with as many numeric characters as possible (of course, there is the limit of your computer memory!). Also, as indicated earlier, you can perform arithmetic operations on integers in various bases. Python converts these integers to decimal and then performs the arithmetic operation.

Example 1: Assuming we need to multiply 52 by 158 . 5 in base 2 is ‘101’, and 15 in base 8 is ‘17’

In [9]:​ # This code multiplies integer in various bases and displays the decimal equivalent

# to specify 101 in base 2, we use 0b101

# to specify 17 in base 8, we use 0o17  

0b101 * 0o17     

Out[9]:​ 75

Exercise: Try out some random mathematical operations with integers in various bases; hexadecimal is still unused.

Floats

These are numbers that have a decimal point in their representation. They can be considered as real numbers as defined in mathematics. Floats were introduced as a means of increasing the precision of representing numbers in computer programs.

All the numeric operations that are possible with integers can also be replicated with floats, and Python also supports float – integer operations.

Python 3 handles float operations in a straightforward manner. In Python 2, however, when doing division, at least one of the numbers (usually the divisor) must always be represented as a float to avoid truncation error in the result.

Recall that integers have unlimited representation in Python? Floats do not have the same. The maximum value a float can take is less than 1.8e308 (which is considered infinity in Python).

In [10]:​ 1.8e308

Out[10]:​ inf

On the other end of the number line, floats have a minimum value of 1e-325 which is considered zero.

In [11]:​ 1e-325

Out[11]:​ 0.0

Tip: Try out all the operations you have done with integers by using floats.

Complex numbers

If you know about complex numbers, chances are: you have some mathematics or engineering background. This shows the potential for applying Python to solving abstract engineering or mathematics problems.

Complex numbers are represented by their real and imaginary parts in Python, and all the rules of mathematical operations that apply to integers and floats are still valid with Python’s complex numbers.

In Python, complex numbers are represented as {real} + {imaginary} j . Although, mathematicians might throw a fit, since they prefer the ‘i’ operator for imaginary number.

Example 2: write a program to calculate the product of 5+3i and 2+4i (engineering problems might have the form: 5+3j and 2+4j or 5+j3 and 2+j4)

In [12]:​ # This program calculates the product of complex numbers

​ ​ (5+3j) * (2+4j)

Out[12]:​ (-2+26j)

Strings

This is the representation of text characters in a programming language. In Python, strings are specified between quotation marks, either single, double or three quotes. Remember the Hello World program we wrote earlier on? Well it was a string!

Example 3: Let us try creating strings with the single, double and three quotes delimiters. You may notice that Python displays the same results.

In [13]:​ ‘Michael Foster’​

Out[13]:​ ‘Michael Foster’

In [14]:​ “David Beckham”

Out[14]:​ ‘David Beckham’

In [15]:​ """Guillermo Giovani"""

Out[15]:​ ‘Guillermo Giovani’

You may wonder, why strings are equally represented by either single, double or three quotes? Why not stick to one convention? The answer is obvious in the way the English language is written. There are instances when written words require either single or double quotes (at most). The rule is to use a higher number of quotes to wrap the string.

For example, let’s say I want Python to print the following statement: Hello Mark, it’s nice to meet you!

Let us try the single quotes:

In [16]:​ ‘Hello Mark, it’s nice to meet you!’

  ​ File "<ipython-input-23-43365d0419fc>" , line 1

 ​    ​ 'Hello Mark, it's nice to meet you!'

                    ^

  ​ ​ SyntaxError: invalid syntax

Using double quotes, however, gives us the right answer.

In [17]:​ "Hello Mark, it's nice to meet you!"

Out[17]:​ "Hello Mark, it's nice to meet you!"

Notice that the output in this case is enclosed in double quotes as well. This is due to the escape of the apostrophe within the expression (an apostrophe is interpreted as a single quote by Python).  The same process follows for double quotes in expression.

While it is possible for three quotes to be used for writing multiline strings/comments. It is not a formal way of doing so. The # is still the default comment approach in Python (line by line).

Example 4: Let us try a fun example. We would display a cool Mortal Kombat logo using the print function. This is just a modification of the ‘Hello world’ program we wrote earlier.

To do this example, we need to have a way to specify the logo as strings. For this, we need ASCII art, which you can design yourself, or copy example art from this repository: https://www.asciiart.eu/video-games/mortal-kombat

The code and the output art can be seen in figure 3.

Figure 3: Illustrating graphics with the print function and strings

Exercise: Visit the Ascii art repository, and try to replicate displaying graphics using Python.

String Concatenation

While it is not feasible to do arithmetic operations with strings (strings are interpreted as text, you cannot do math with text), strings can be joined together or replicated.

Example 5: Let us assume we want to write a random person’s name. We have the First and Last name.

In [18]:​ ‘Fire’ + ‘ cracker !’

Out[18]: ​ ‘Fire cracker’

Tip: In this example, notice how I introduced a space before the word ‘cracker’. This is to ensure the result makes sense, as otherwise Python would have strung everything together like this ‘Firecracker’ .

Example 6: Now let us try to replicate a string. Imagine you need to type the same sentence 100 times! Only if there was a way to do it just once?

In this example, we would repeat the statement ‘I love Python programming, it’s awesome’, three times. The same method used can be adapted for a larger repetition, it only requires your creativity!

In [19]: ​ print(“I love Python programming, it’s awesome\n” * 3)

Out[19]: ​ I love Python programming, it’s awesome

​ ​ I love Python programming, it’s awesome

​ ​ I love Python programming, it’s awesome

Tip: The print function helps to display strings in a proper manner, without quotes. Also, it allows the use of formatting options, like the new line character \n in the code above. Try using this tip to display more elaborate strings.

Variables

These are placeholders for data of any type i.e. integers, strings, etc. While it is very effective to imagine a variable as a storage space for temporarily keeping data until it is needed, I prefer to think of variables like money.

Wait what? Yes, Money. While it doesn’t technically equate in description, it is good for visualization of what a variable does with your data.

First, I believe you are aware of the fact that money only has the value to which a consensus of people, say the world bank, assign to it! (it is more complicated than that, but that is the general idea). That means the $10 note you have today can either be more valuable or less valuable in the next year depending on what happens in the economy of the issuing country, and across the world. It could also be worth nothing if the world so decides.

Now, relating it to variables. As earlier described, they are placeholders for data. To declare a variable, say “X”, you will assign data with the value of a specific data type to it. Then, the value assigned to it becomes the value of your variable. Now, the variable carries the same value and datatype as the original data until you choose to re-assign it, or it gets re-assigned by events in your program.

Makes sense? Let us try a few examples to get a better understanding of variables in Python.

In [20]: ​ x = 5

            y = 10

            x + y

Out[20]: ​ 15

As you may observe from the code, x is now of value 5 and y of value 10. If we check the data type of x or y, they would both have the same int() datatype as their assigned data.

In [21]: ​ type(x)

Out[21]: ​ int

Exercise : Try checking the type for the other variable y. Change the values of x and y and perform extra operations with them. Remember that variables can hold strings as well, just remember to put the quotes for declaring strings.

Tip : There are a few rules for declaring Python variables: First, Python variables can only begin with an underscore (_) or an alphabet. You cannot start a Python variable with numbers or special characters i.e. @, $, %, etc.

Second, Python has some reserved names for functions i.e. print, sum, lambda, class, continue, if, else, etc. These names cannot be used for declaring variables to avoid conflicting Python’s operations.

Also, in variable declaration, it is advisable to use a naming convention that best describes what your variable does. If your choice variable-name contains multiple words, separate them with an underscore like this: variable_Name, matric_Number, total_Number_Of_Cars, etc.

Example 7: let us build a more elaborate program to illustrate string concatenation.

In [22]: ​ # This program concatenates first and last name and displays fullname.

​   ​ First_name = ‘James’

​ ​ Last_name = ‘Potter’

​ ​ Fullname = First_name + ‘ ‘ + Last_name

         ​ print (Fullname)

​    

Out[22]: ​ James Potter

Tip: Did you notice there was no use for quotation marks in the print function? This is because the variable ‘Fullname’ already has the quotes as its attributes. By putting strings in the print function, Python would think you wish to print the string ‘Fullname’ !

Let us check the attribute of the variable ‘Fullname’.

In [23]: ​ type(Fullname)

Out[23]: ​ str

Again, let us build a more complicated version of the name program that accepts user input for first name, surname and age, and then displays something witty.

In this example, the function input() would be used, and its syntax can be deduced from the code. 

In [24]: # This program displays name and says witty stuff

  ​ First_name = input ('What is your first name ?\t')

  ​ Last_name  = input ('Last name too ?\t')

  ​ Age        = str (input ('Your age ?\t '))

  ​ Fullname = First_name + ' ' + Last_name

  ​ print ('Hello '+ Fullname + '\n\n')

  ​ print ('It must be nice to be '+ Age + '; I am a computer, I     have no age!')

Out[24]: ​ What is your first name ?​ David

​ ​ Last name too ?​ Maxwell

 ​    ​ Your age ?​ 25

   ​ ​ Hello David Maxwell

​ It must be nice to be 25; I am a computer, I have no age!

It might also be easy to observe the datatype conversion of the age variable to a string using the str () function. This is also possible for conversion to other data types i.e. int () for conversion to integers, float () for conversion to floating-point numbers, and complex () for changing to complex numbers.

Exercises

Create a simple program that takes the name of three friends, and welcomes them to a restaurant e.g. McDonalds.

Write a program that accepts an integer value and displays it as a floating-point number.

Write a simple program that accepts a temperature value in Celsius, and displays its Fahrenheit equivalent.

Hint: Conversion from Celsius to Fahrenheit is given by:

Lists

Python lists are not so different from the general idea of lists in regular language. It is just a collection of values in a sequence separated by commas. Lists can hold values with different data types i.e. a Python list can be a combined sequence of integers, floats, strings etc. Lists can be assigned to variables at their declaration, and a few list operations are illustrated in the following examples:

# List examples

[1,2,3,4,6]                   # List of integers

[12.1,13.42,15.6,18.93,20.0]  # List of Floats

['New','Taken','Extra']       # List of strings

['True','False']              # List of Boolean expressions

['Derek',25,125.50,True,]     # List of different data types

To declare a Python list, the square brackets [ ] are used, and values are separated by commas.

All the lists declared above show how Python lists are flexible for holding data of any type. However, these lists are just declared to be evaluated once at the command window. It is a better practice to assign lists to variables for re-use.

Tip: To run the above example, copy and run each list individually, if you copy all these lists above into a Jupyter cell, Python will return only the result of the last declared list.

List indexing

Once you have declared a list, each value in the list can be accessed via a method called indexing. To understand this concept, imagine a list is a library with various types of books. Librarians usually organize books by category/type, and as such, you wouldn’t expect to find a science book in the arts section. Also, in each category, the librarian can ease access by allocating a serial number to each book. For example, the book on ‘World geography’ could have the identifier, ‘Science A2’ which belongs only to that book in that specific library. In another library, that same identifier can refer to ‘Space science’ or something else. The point is, each element in a specific list has an identifier for accessing it, and list indexing is the process of retrieving elements of a list using their identifiers.

To use list indexing, the list must first be assigned to a variable. It is also worth noting that Python list indexing is direction dependent. For indexing from left to right, Python starts indexing at ‘0’, while right to left indexing starts at -1.

Example 8: Let us grab the third element from a list.

In []:​ Random_List = [1,2,3,4,’food’]

 ​ ​ Random_List [2]

Out[]: ​ 3

Alternatively, we could have grabbed the value 3, from Random_List by using the command Random_List [-3] ; however, it is not a common convention.

There are also other versions of lists that contain other lists. These are called nested lists.

Example 9 : This is a Nested list

Random_List2 = [[1,2,3,4,'integers'],'food', [12.3,5.2,10.0,'floats']]

To understand nested lists, imagine Russian nesting dolls. The name is derived from the fact that the first doll layer contains the other dolls. If the layer is removed, the next layer then houses the rest of the dolls. This goes on until all the layers of dolls are removed to reveal the desired layer of the doll. What this analogy portrays is that the same indexing approach works for lists that are nested within other lists.

Example 10: Let us grab the value 10.0 from Random_List2.

In []:  Random_List2 = [[1,2,3,4,'integers'],'food',[12.3,5.2,10.0,'floats']]

  ​ Random_List2 [2][2]

Out[]: 10.0

Here is what happened. By using the first command, Random_List2 [2] , we were able to grab the nested list containing the desired value 10.0. If you count, that list is the third element in Random_List2 . Now, after using the first indexing to grab that list, we can now index that new list as well. Within this new list ‘Random_List2[2]’ the value 10.0 is the 3 rd element, or at index 2 (remember that Python indexes from zero, we could also select it with ‘-2’ ), so to grab it, we just index that list at 2: ‘Random_List2[2][2]’ . Easy enough? You can try grabbing other elements of the list using this idea.

Example 11: Let us create a program that accepts user information and displays some results.

In []: ​ # Program that accepts user data

  ​ user_Data = eval (input ('''Please enter your information in the following order:

  ​ ['Name', Age, Height (in metres), Married (enter True/False)]

'''))

Please enter your information in the following order:

['Name', Age, Height (in metres), Married (enter True/False)]

['James Franco', 41, 1.8, False]

Output

Te ‘eval’ method in the code above is used to evaluate any input suggested by the user. This allows Python to accept the input in literal string form and interpret it as Python code. In this instance, Python recognizes the input as a list because of the [ ]. It also assigns the appropriate datatype to each element as specified in the list.

Now that the user_Data list has been prepared, we can now assign each element in the list to their corresponding variables and print our desired output.

In []: ​ Name ​ ​ = user_Data[0]

         ​ Age       ​ = user_Data[1]

          ​ Height    ​ = user_Data[2]

          ​ M_Status ​ = user_Data[3]

          ​ print(('''Here are your details

      ​ Name: {}

       ​ Age: {}

       ​ Height: {}m

       ​ Married: {}''').format(Name,Age,Height,M_Status))

Output

Here are your details

Name: James Franco

Age: 41

Height: 1.8m

Married: False

Now, that was fun right? However, you may notice that a new method was introduced in this section, the ‘. format’ print method. Here is how the code works: From the user_Data list we have declared in the first cell, we grabbed each element and assigned them to variables ‘Name, Age, Height and M_Status’. To display all these elements without the need for concatenating strings or changing data types, we can use the print function with triple quotes to print across multiple lines. Alternatively, we could have used single or double quotes along with newline/tab escape options i.e. \n or \t.

The ‘. format’ method is a way of controlling print. You just need to put { } in the places your values would go, and then specify the order.

Assuming we want to print something like: I am 5 years old, and I have $20.

In []: ​ print (('I am {} years old, and I have {}.').format(5,'$20'))

Output

I am 5 years old, and I have $20.

Hint : In the code above, the $20 had to be specified as a string. A better approach would be to use variable assignment i.e. Assigning 5 and $20 to variables and passing them into the ‘.format’ method.

Exercise: Write a program that displays a customer’s bank account history. This is just a variation of the example program. If your code does not work, review the examples, hints and comments to get a new insight.

Indexing a range (List slicing)

Assuming we need to grab a range of elements from a list, say the first three elements, this can be achieved via range indexing or slicing. The range operator is the colon ‘:’, and the technique has the following syntax:  List name [range start : range end +1].

Example 12: Grab the names of domestic animals from the following list [‘Cat’, ‘Dog’, ‘Goat’, ’Jaguar’, ’Lion’] and assign it to a variable domestic_animals .

In []:         Animals = ['Cat', 'Dog', 'Goat', 'Jaguar', 'Lion']

               domestic_Animals = Animals [0:3]

               print(domestic_Animals)

Output

['Cat', 'Dog', 'Goat']

Alternatively, if our desired data starts from the first element, there is no need for specifying the index ‘0 ’ (or index ‘-1 ’ in right to left indexing), we could have also grabbed the data with ‘domestic_Animals = Animals [: 3]’ .

You may observe that the element ‘Goat’ is at the index ‘2’, however, the range syntax requires the ‘range end’ argument to be specified as ‘range end +1’ which is 3 in this case.

The same approach is used for doing more complicated data grabbing as seen in the next example.

Example 13: Write a program that selects three different treats from a list of food and prints a message.

In []: food_list = ['rice', 'salad’, ['cake’, ‘ice-cream', 'cookies', 'doughnuts'],'Beans']

  ​ treats = food_list [2] [1:4]

       print ('I love {}, {}, and {}.’.format(treats[0],treats[1],treats[2]))

Output

I love ice-cream, cookies, and doughnuts.

See how we combined the range indexing with the .format method?  Try out some fun examples for yourself and consider your proficiency in using these techniques.

String Indexing

Strings are just a bunch of characters ‘strung’ together. Hence, all list indexing operations are also applicable to strings.

Example 14 : Grab the word ‘except’ from a string.

In []:​ # This program illustrates string indexing

String = 'Exceptional'

# index -slice

new_string = String[:6]

# Printing the output

print(new_string)

Output

Except

Example 15: Grab the word ‘oats’ from any element in a list.

In []: ​ # grabbing the word 'medical' from a list

  ​ word_list = ['nautical',['medieval', 'extravaganza'],'extra' ]

  ​ word = word_list[1][0][:4] + word_list[0][5:8]

  ​ print(word)

Output

medical

Tip : keep trying out variations of list and string indexing, they will be very useful in subsequent chapters.

Now that we have learned how to index lists and strings, let us consider the case where we need to assign new elements to a list.

The first case is element re-assignment. Lists are mutable (strings are not mutable), which means they can allow you to change their initial declaration. You can re-assign list elements using indexing, and you can also add or remove from a list using the .append , .insert , .extend , .remove methods.

Example 16: Here is a list of fruits we want to manipulate [‘Apple’, ‘Orange’, ‘Banana’, ‘Cashew’, ‘Almond’].

In []: # Declaring the list of the fruits

       fruits = ['Apple', 'Orange', 'Banana', 'Cashew', 'Almond']

       fruits

Out[]: ['Apple', 'Orange', 'Banana', 'Cashew', 'Almond']

Now, let us re-assign the second element in the list i.e. ‘Orange’ and change it to ‘Guava’

In []: fruits[1] = 'Guava'

fruits

Out[]: ['Apple', 'Guava', 'Banana', 'Cashew', 'Almond']

Notice that guava has been indexed as the second element in the list. However, Orange has been removed/replaced.

Hint: This method uses the idea of variable re-assignment. The element ’Orange’ is a variable in the fruits list and its name is based on its index i.e. fruits[1], just as the variable name for ‘Banana’ would be fruits[2]. So, it's just a matter of re-assigning a value to the variable like we learned in the previous section.

The same method can be used to re-assign any elements in that list. However, what if we have a nested list?

Example 17: Nested list re-assignments also follow the same idea. First grab the nested element to be re-assigned, and then assign a new value to it.

In []: # To re-assign a nested element.

  ​ New_list = ['Apple', ['Orange','Guava'], 'Banana', 'Cashew', 'Almond']

  ​ New_list

Out[]: ['Apple', ['Orange', 'Guava'], 'Banana', 'Cashew', 'Almond']

In []: New_list[1][0] = 'Mango'

       New_list

Out[]: ['Apple', ['Mango', 'Guava'], 'Banana', 'Cashew', 'Almond']

Exercise: Now, try changing the element ‘target’ to ‘targeting’ in this list:

nest = [1,2,3, [4,5, ['target']]]

Using our initial list of fruits, let us use the .append method. This method takes the value passed to it and adds it to the end of the list. It can only add one element to a list at a time. If more than one element is passed into the method, it adds it as a sub-list to the intended list.

Example 18: Add the fruit Pawpaw to the list of fruits and make it the last element.

In []: # Declaring the list of the fruits

        fruits = ['Apple', 'Orange', 'Banana', 'Cashew', 'Almond']

​ fruits.append ('Pawpaw')

​ fruits

Out[]: ['Apple', 'Orange', 'Banana', 'Cashew', 'Almond', 'Pawpaw']

To add a sub-list, use the method  .append (‘[sub-list elements]’)

Example 19: Let us add a sub-list of carbohydrates to this food list

In []: #Declaring the food list

       foods = ['Beans','plantain','fish']

  ​ foods.append (['Rice, 'Wheat'])

​ foods

Out[]: ['Beans', 'plantain', 'fish', ['Rice', 'Wheat']]

Now, let us use another method ‘.extend’ . This method also adds its input values to the end of a list. However, unlike the .append method, in the case of multiple input values, the .extend method adds each element to the list, thereby extending the list index.

Example 20:   Use the .extend method to add two elements to the end of a list.

In []: # Declaring the food list

         foods = ['Beans','plantain','fish']

         foods.extend(['Rice', 'Wheat'])

   foods

Out[]: ['Beans', 'plantain', 'fish', 'Rice', 'Wheat']

Tip: The .append and .extend methods both accept just one input argument. If it is more than one, enclose the inputs in a square bracket. This is seen in examples 19 and 20.

Good, you have learned how to add elements to the end of a list using the .append and .extend methods. Imagine a case where you want to add an element to any part of a list without replacing the element at that index. The .insert method is appropriate for this. It also accepts one argument, and inserts that value at the specified index.

The .insert method has the following syntax:  ListName.insert (Desired index, New value)

The ListName is the name of the list to be manipulated. Desired index is the expected position of that element (remember indexing starts from 0, or -1 depending on indexing preference), and New value is the element that is to be inserted at the specified index.

Example 21: Consider the following list of integers, add the right value to make the sequence correct. Count_to_ten = [1,2,4,5,6,7,8,10].

In []: # This program illustrates the .insert method.

         Count_to_ten = [1,2,4,5,6,7,8,10]

         Count_to_ten.insert(2,3)

         Count_to_ten.insert(8,9)

         Count_to_ten

Out[]: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Let us review how that worked. The first use of the .insert in example 21 assigns the value 3 to index 2 as specified by the syntax. Now the list ‘Count_to_ten’ has increased in size and indexing by a value of 1 i.e. Count_to_ten = [1,2,3,4,5,6,7,8,10] . Now we can add the value 9 to the proper index of 8 as shown in the 4th line of code.

Tip: Be aware of the index expansion before assigning elements. In the previous example, if we assigned the value 9 first, the indexing of the other value 3 would not have been affected by the increase in the list size. Then the following code would have achieved the same result:

In []:   Count_to_ten = [1,2,4,5,6,7,8,10]

         Count_to_ten.insert(7,9)

         Count_to_ten.insert(2,3)

         Count_to_ten

Out[]: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

This is because the insertion of a value in the list shifts the list elements from that index of insertion to the right. Each of the shifted elements now has an index incremented by the number of elements inserted i.e. if an element had an index of 3 in a list, and two items were inserted at its index or before its index, that element now has an index of 5.

The last method we would consider on list operations is the .remove   method. This is used for removing an element from a list. It also has a similar syntax with the .insert method and uses list indexing.

Example 22: Let us add the value 11 to the list Count_to_ten and then clear it using the .remove method.

In []: # Using the output list from the last example

  ​ Count_to_ten.append(11)

Out[]: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10,11]

In []: # Resetting back to 10

  ​ Count_to_ten.remove(11)

Out[]: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

There is something to notice here, the .remove method does not require the specification of the index of the unwanted element. Only the name or value of the unwanted element is needed.

Exercise: Create a list of cars, and practice various manipulations on it. Practice adding to the list, removing elements etc.

Hint: let your exercise require all the list manipulations we have practiced so far. Get creative!

Recall how lists are similar to strings in some operations? Well, you can also do list addition and multiplication (repetition) as we did with strings.

Example 23: Here are a few list additions and multiplications

In []: # Addition and multiplication of lists

  ​ List1 = [‘A’, ’B’, ’C’, ’D’]

  ​ List2 = [1, 2, 3, 4]

  ​ CombinedList = List1 + List2

  ​ CombinedList

Out[]: [‘A’, ’B’, ’C’, ’D’, 1, 2, 3, 4]

In []: # Multiplication of lists

  ​ List2*2

Out[]: [1, 2, 3, 4, 1, 2, 3, 4]

While there are more operations possible with lists, we have covered enough to serve as a good foundation for further applications with lists.

Tuples

These are similar to lists i.e. a sequence of comma separated values. However, unlike lists, tuples are immutable and they are defined using a parenthesis () . Data can also be grabbed from tuples using the indexing technique. The usefulness of tuples arises from the need to create program parts that cannot be changed by the user/program during operation. Imagine creating a bank account program. Things like your ATM pin could be open for user modification. However, imagine letting people have access to change their account details! This could cause a conflict in the banking program. These are the kinds of data that tuples can hold. Users can retrieve such data; however, they cannot be changed.

Example 24: Let us declare some tuples.

In []: # Tuples can be declared in two ways

  ​ my_Tuple = (1,2,3,4,5) ​ # way 1

   ​ print(my_Tuple)

   ​ type(my_Tuple)

Out[]:   ​ (1,2,3,4,5)

​ tuple

In []:  my_Tuple2 = 1,2,3,4,5, ​ # way 2

   ​ print(my_Tuple2)

   ​ type(my_Tuple2)

Out[]: ​ (1,2,3,4,5)

​ tuple

We can consider the two types of tuple declarations shown in the codes above. While the first method is the traditional and recommended method, the other method is equally acceptable. The second method is called ‘tuple packing’, and we would discuss its relevance later.

Any sequence of comma-separated items declared without an operator or special characters like [ ], { }, etc. automatically gets assigned as a tuple. This is important.

As explained earlier, all the regular indexing operations involving data grabbing can be done with tuples, however, it doesn’t support element assignment/re-assignment once declared.

Another reason why a Python programmer might prefer to use a tuple over a list is because tuple operations are faster than with lists as tuples require less memory allocation.

Example 25:

In []: # Comparing Lists with tuples

  ​ my_List = ['Men', 'Index', 1,2,3]

  ​ my_tuple = ('Men', 'Index', 1,2,3)

  ​ a = my_tuple.__sizeof__() # get tuple size

  ​ b = my_List.__sizeof__()  # get list size

  ​ print('size of list is {}, and size of tuple is {}.'.format(b,a))

Out[]: size of list is 80, and size of tuple is 64.

Even though the elements in a tuple and list are exactly the same, they have different sizes (in terms of memory). This is especially useful for designing systems that have limited resources with high-efficiency requirements.

Example 26:  Let us grab some elements from a tuple using indexing

In []: # grabbing tuple elements

  ​ A = my_tuple[0]

  ​ B = my_tuple[1]

  ​ print ('The {} of {} is 0.'.format(B,A))

Out[]: The Index of Men is 0.

Exercise: Try out other indexing operations with tuples. Create a tuple and try to re-assign one of the elements using indexing. What did you observe and why?

Tuples also allow an operation called ‘packing and unpacking’. Either operation is dependent on the side of the equality operator the tuple is.

Example 27: Let us illustrate tuple packing and unpacking.

In []: # tuple packing and unpacking

  ​ Tuple = 'Daniel', 'Dean', 'James'   # Tuple packing

  ​ [Name1, Name2, Name3] = Tuple        # tuple unpacking

  ​ print(Name1)

  ​ print(Name2)

  ​ print(Name3)

Out[]: Daniel

  ​ Dean

  ​ James

Tuple packing and unpacking are kind of like a convenient way of using tuples for assignment/creating variables. Packing a tuple is just a matter of declaring the tuple, to unpack however, you just declare a number of choice variables equal to the number of elements in the tuple. Each variable then takes the corresponding value in the tuple according to the order of arrangement. As can be observed, the Name1 variable corresponds to the ‘Daniel’ tuple element, and so on.

Hint: The square brackets around the declared variables is unnecessary in tuple unpacking. It is just standard practice to use, however, using () or nothing will give the same results. Try it!

A common use for tuples is to hold value pairs. This is especially useful for collecting data like user information for which each element stores a specific user data.

Example 28:

In []: # collecting data with tuples

  ​ User_Info = (('Name','James'),( 'Age',22))                                               #Nested tuple

  ​ A,B = User_Info[0] ​ # Inner Tuple unpacking

  ​ C,D = User_Info[1] ​ # Inner Tuple unpacking

  ​ print (A, ': ',B)

  ​ print (C, ': ',D)

Out[]:  Name :  James

​ Age :  22

Dictionaries

These are also similar to lists and tuples in certain ways. Unlike lists and tuples, dictionaries are declared as key-value pairs and by using the curly braces i.e.{ } . Think of dictionaries like safety deposit boxes. You can keep any item you want in it (except your car of course, size does matter), but you can only access it with your key!

In Dictionaries indexing is done with the key associated with the value stored. Dictionaries are mutable in their keys and values (not-so-safe-deposit-box), however, they can be secured by using tuples as their keys or key-value pairs (depending on access level desired).

Example 29: Let us declare a dictionary.

In []:  # Dictionary declaration

   ​ my_Dict = {'Key1': 'Movies', 'Key2': ['Iron Man', 'Avengers']}              

   ​ print(my_Dict)

   ​ print(type(my_Dict))

Out[]:  {'Key1': 'Movies', 'Key2': ['Iron Man', 'Avengers']}

  ​ <class 'dict'>

There, we just declared a dictionary. Notice that the first value is a string, and the second value is a list. The keys can be other datatypes as well, and not necessarily strings; however, string key names are usually preferred for keeping values for easy call and code readability.

To grab elements in a dictionary, the keys holding the value is first called.

Example 30: Grabbing dictionary elements

In []: A = my_Dict['Key1']

  ​   B = my_Dict['Key2']

  ​   print (A,': ',B)

Out[]: Movies :  ['Iron Man', 'Avengers']

We can see that by calling the dictionary keys, we passed the values held by Key 1 and key 2 to the variables A and B. More complicated dictionary key indexing can be done.

Example 31: Grabbing elements from a nested dictionary.

First, we declare our dictionary:

In []: Acct_Dict = {'Name':'Customer1','Account type'

  ​ ​ :{'type1':'Savings','type2': 'Current'}}

       Acct_Dict

Out[]: {'Name': 'Customer1', 'Account type': {'type1': 'Savings', 'type2': 'Current'}}

If we need to print that the user has a savings type of bank account, we would need to grab the Second key ‘Account type’ first, then grab the first key of that resultant dictionary. Then we can easily pass that into the print statement.

In []: print('You have a '+ Acct_Dict['Account type']['type1']+ ' account')

Out[]:  You have a Savings account

Notice how the word ‘Savings’ has been grabbed using dictionary key-value indexing. Since the name of the keys for a dictionary is important to getting its values, it is important to determine the keys it holds at any point in time. To view a list of the keys a dictionary holds, use the .keys() method.

In []: Acct_Dict.keys()    # Checking keys held by Acct_Dict

Out[]:  dict_keys(['Name', 'Account type'])

In []: Acct_Dict['Account type'].keys()    # Checking the inner keys

Out[]:  dict_keys(['type1', 'type2'])

Tip: There are other methods that can be used with a dictionary, even though this is all we would need for our lessons here. However, to use these methods with a dictionary or to find the methods available for any other datatypes, use the dir() method.

In []: # Checking possible methods with datatypes

  ​ # Let us declare some datatypes first and check their methods

  ​ String = 'Bob'; integer = 100; Float = 25.3

  ​ List = ['Man']; Tuple = 5,   # To declare a single tuple

        # put a comma   after the single value

  ​ Dictionary = dict([('Name','Max')])           # Dictionaries can be declared

        # using dict() as well.

  ​ A = dir(String); B = dir(integer); C = dir(Float)

  ​ D = dir(List); E = dir(Tuple) ; F = dir(Dictionary)

  ​ print (''' Here are the methods possible with each type

  ​ Strings

  ​ {}

  ​ Integers

  ​ {}

  ​ Floats

  ​ {}

​ Lists

  ​ {}

  ​ Tuples

  ​ {}

​ Dictionaries

  ​ {}

  ​ '''.format(A,B,C,D,E,F))

Try copying and running the code above, or write your own variation. However, the point is for you to find out all the possible methods available to any data type in Python. This information gives you access to do advanced manipulations. Also notice the comments in the code, some new methods of declaring ‘Tuples’ and ‘Dictionaries’ were introduced. These are extra; just tricks you may wish to use. Same with the use of semi-colon to allow for declaring multiple variables or commands in one line.

Booleans

These are conditional datatypes that are used to determine the state of a statement or block of code. Booleans have two possible values ‘True’ or ‘False’ which have corresponding integer values of 1 and 0 respectively.

In []: ​ A = True   # Boolean values seem like strings but do not enclosed in quotations

  ​ type(A)

Out[]: ​ bool

In []:​ int(A)   # Integer value of the Boolean

Out[]:​ 1

As can be seen, the corresponding integer value for Boolean ‘True’ is 1. Try the above code for the Boolean ‘False’.

For most operations, Booleans are usually output values used for specifying conditions in a loop statement or checking for the existence of an element or condition. This leads us to the use of comparison and logical operators.

Comparison operators

These operators, as their name implies, are used to check for the validity or otherwise of a comparison. The following are comparison operators:

Table 1: Comparison operators

<

>

==

<=

>=

!

in

Less than

Greater than

Equal to

Less than or equal to

Greater than or equal to

Not operator (can be combined with any of the above)

This checks for the existence of an element/value

Example 32:   Let us check for conditions using the comparison operators

In []: #This code illustrates the Boolean comparison output

  ​ print (5 < 10); print (3>4); print('Bob'=='Mary');

  ​ print (True == 1); print (False == 0); print (True != 1)

Out[]: True

  ​ False

  ​ False

  ​ True

  ​ True

  ​ False

In []: #The ‘in’ operator

  ​ print ('Max' in 'Max Payne'); print (2 in [1,3,4,5]); print (True in [1,0])

Out[]: True

  ​ False

  ​ True

Logical operators

These are the Python equivalent of logic gates. This is a basic technique for performing logical operations and can also be combined using the interpreted versions of De Morgan’s law.

In case you were wondering, De Morgan is a British guy that found a way to simplify Boolean logic using rules earlier invented by another British guy, George Boole (Boolean is named after this guy).

Table2: Logical operators

and/&

or/ |

not/ !

Evaluates to true if, and only if, both operands are true, false otherwise.

Evaluates to true if, at least, one of the operands is true, false otherwise.

This is an inversion of the value/operation of its operand.

Example 33: Let us try out some logical operations (truth table) with the ‘and’ operator .

In []: #The ‘and’ operator

  ​ print (True & True); print (True and False);print (False and False)

  ​ print (False and True); print('foo' in 'foobar' and 1<2)

Out[]: True

  ​ False

  ​ False

  ​ False

  ​ True

Example 34: Let us try out some logical operations (truth table) with the ‘or’ operator.

In []: #The ‘or’ operator

  ​ print (True | True); print (True or False); print (False or False)

  ​ print (False or True); print (('foo' in 'foobar') | 1<2)

Out[]: True

  ​ True

  ​ False

  ​ True

  ​ True

For the last operation in the above code, notice how the string operation is enclosed in parenthesis before the ‘logical or’ is used. This is because Python throws an error in the case of comparing string operations directly with other types using the logical or.

Exercises:

Here are two food lists: [‘Beans’,’Wheat’,’Bread’] and [‘Rice’,’Plantain’,’Pizza’,’Spaghetti’]

Write a Python code to check if the word ‘Rice’ exists in both lists.

Write a Python code to check if ‘Pizza’ exits in at least one of the lists.

Create a Python dictionary with two keys. Key 1 should be immutable, while key 2 can change. Alternatively, the values held by key 1 and 2 are a sequence of integers; however, the values of key 1 can be edited, while that of key 2 should be immutable. Hint: Recall, immutable data types are strings and tuples.

Now that we have gone through all the data types, let us move on to loops and conditionals.

Conditional statements and Loops

These are a little different from what we have been doing so far. Our previous examples can be described as ‘sequential program execution’, in which expressions are evaluated line by line without any form of control. Conditional statements, however, are used to take control of how and when lines of code are executed. Loops on the other hand, are used to repeat the execution of a specific code or blocks of code. These two different control algorithms are mostly used together to develop programs of varying complexity, although, they can be used independently of each other.

Let us consider the most basic and often used conditional statement – the ‘IF ’ statement.

This has a syntax of the form:

                    if  expression :

                              statement

The expression in this case is usually a Boolean operation, while the statement is a line/block of code to be executed once the Boolean evaluates to either true or false (or is not true or false).

Example 35: Let us write a program that grants a user access when any of three correct passwords is entered.

In []: # Grant access if user password is correct

  ​ password_pool = ('Smith_crete','Alex@456','CEO4life')

  ​ user_password = input ('Please enter your password: \t')

  ​ # Now the IF condition

  ​ if user_password in password_pool:

  ​ print('\n Access granted!')

Out[]: Please enter your password: CEO4life

  ​ Access granted!

That was a fun program, right? The idea is for Python to check if the password entered by the user is available in the earlier declared password pool (notice that the password pool is a tuple? This is a practical way for creating fixed/secure passwords)

Exercise: Write a program to calculate how much a person owes for keeping a movie past rent-due-date. Let there be an increase in price given the person keeps the movie past 3 days.

Example 36:

In []: # Movie rent-overdue price

  ​ price1 = 5      # $ 5 for every day past due within first 3 days

  ​ price2 = 7      # $ 7 for every day past due after first 3 days

  ​ days_past_due = eval(input('How many days past due:\t'))

  ​ # If statement

  ​ if days_past_due <= 3:

     ​ ​ print('\nYou owe $',(days_past_due*price1))

  ​ if days_past_due > 3:

   ​ ​ print('\nYou owe $',(3*price1 + (days_past_due - 3)*price2))

Out[]: How many days past due: ​ 4

  ​ You owe $ 22

See how we can combine multiple if statements to write even more complex code? However, there is a better syntax for evaluating multiple conditional IF statements; the IF-ELSE statements.

It has a syntax like this:

                    if  expression :

                              statement

​ ​   else:

                        alternative statement

Example 37: Improved version of example 35 using IF-ELSE

In []: # Grant access if user password is correct or raise alarm

  ​ password_pool = ('Smith_crete','Alex@456','CEO4life')

  ​ user_password = input ('Please enter your password: \t')

  ​ # Now the IF-ELSE conditions

  ​ if user_password in password_pool:

  ​ ​ print('\n Access granted!')

  ​ else:

​ ​ print('\n Access Denied! Calling Security …')

Out[]: Please enter your password: ​ wrong password

  ​ Access Denied! Calling Security …

Now, this seems like a more reasonable security system, right? Still, the IF conditional gets better with the IF-ELIF-ELSE syntax. This allows you to specify actions in multiple situations.

Syntax:

                   if  expression :

                              statement

                   elif  expression :

                              statement

                   elif  expression :

                              statement

​ ​   else:

                        alternative/default statement

The syntax above can be explained in regular language as “If the first condition is met, then execute the action within the first statement, or else, if the first condition is not met, do the action under the second statement.” This goes on until the else statement that defaults to the inability of the program to fulfill any of the conditions within the IF and ELIF statements.

Example 38: An even better version of example 35.

In []: # Grant access and greets user if the user password is correct.

  ​ # Otherwise, raise an alarm.

  ​ password_pool = ('Smith_crete','Alex@456','CEO4life')

  ​ user_password = input ('Please enter your password: \t')

  ​ # Now the IF-ELIF-ELSE conditions

  ​ if user_password in password_pool and user_password == 'Smith_crete':

  ​ print('\nAccess granted! Welcome Dr.Smith')

  ​ elif user_password in password_pool and user_password == 'Alex@456':

  ​ print('\nAccess granted! Welcome Mr. Alexander')

  ​ elif user_password in password_pool and user_password == 'CEO4life':

  ​ print('\nAccess granted! Welcome Mr. CEO')

  ​ else: print('\nAccess Denied! Calling Security …')

Out[]: Please enter your password: wrong password

  ​ Access Denied! Calling Security …

Out[]: Please enter your password: Alex@456

  ​ Access granted! Welcome Mr. Alexander

Out[]: Please enter your password: Smith_crete

  ​ Access granted! Welcome Dr. Smith

Out[]: Please enter your password: CEO4life

       Access granted! Welcome Mr. CEO

Exercise: Now that you have learned the IF conditional statement and its variations, try out some more creative and complicated examples on your own. For example, you can write a program that checks a database, prints students math exam scores and comments on whether they passed the exam or not.

Hint: The database in your example could be a variation of the ‘password pool in example 37’

Let us now look at loops. We will consider two loops: The While loop and the For loop .  As described earlier, a loop will run indefinitely until a specified condition is met. Due to this, they can be used for automating and performing really powerful operations spanning a large range; however, they must be used with caution to avoid infinite loops!

WHILE loop

This loop is used to execute a set of statements or code as long as a specified condition is true. The conditional statement that controls the algorithm is called a flag and is always true for all non-zero values. When the flag becomes zero, the while loop then passes on to the next line of code following it. There could be a single statement or multiple within the while loop, and Python supports the else statement for the while loop.

A while loop has the following syntax:

while flag :

    statement

Example 39: Writing a while loop to print a statement 5 times.

In []: # while loop to print output 5 times

  ​ i = 1                # counter

  ​ while i < 6:         # flag

    ​ ​ print (i,': I love Python')

     ​ i = i+1        # This Increments the value of ‘i’

 else:

     ​ ​ print ('\nThe program has completed')

Out[]: 1 : I love Python

  ​ 2 : I love Python

  ​ 3 : I love Python

  ​ 4 : I love Python

  ​ 5 : I love Python

  ​ The program has completed

Okay, the program worked. But how?

The first thing is the flag. The variable ‘i ’ was declared with a value of 1. For the first iteration, Python checks if 1 is less than 6: the flag is true since 1 is less than 6, and the print statement runs (along with the counter we added to print the status of the loop). This continues until the loop is complete (when i=6), then the else statement runs (since the flag is now false). Assuming the else statement was absent, the while statement ends (or in the case of a larger program, it passes on to the next line of code).

Hint: Notice the increment we included after the print statement? This is important, as it is what makes the value of our counter change to allow for a specific loop duration. It also had to be placed within the While loop such that the increment executes at each iteration. If we remove the increment code, we would have an infinite loop as ‘i’ (with an initial value of 1) will never be equal to 6

Python loops have some extra control statements for handling their execution. We have the ‘Break’, ’Continue’ and ‘pass’. The break statement is used to stop the execution of the loop once invoked. It could sometimes be useful for safeguarding a loop against defaulting to an infinite loop. The continue statement, however, re-tests the condition of the loop from when it is invoked, this leads to a continuation of the loop from a stop point (usually used to resume the loop after a break statement).

We would not use the pass statement here, as it is only useful for skipping the execution of a loop expression which is otherwise required by the loop syntax.

Example 40: Let us use a while loop along with the continue statement to create a program that prints numbers from 1 to 10, but skips 8.

In []: # This program prints 1-10 but skips 8

  ​ i = 0                # Counter

 ​ while i<=9:

     ​ ​ i = i+1         # Increment

     ​ ​ if i == 8:

         ​ ​ continue

     ​ ​ print(i)    

  ​ else: print ('\nThe program has ended')

Out[]: 1

 2

 3

 4

 5

 6

 7

 9

 10

 The program has ended

Tip: Notice how our increment was placed before the ‘ continue statement’? This is important to prevent an infinite loop. In this case, when the counter ‘ i ’ equals 7, it is incremented to 8 and the if statement executes the ‘ continue statement’. The condition is then re-tested, and the increment adds 1 to the value. Since 9 is greater than 8, the rest of the code runs till the loop ends.

Assuming the increment is after the if & continue statements, when ‘i’ is equal to 8, the continue statement re-tests the condition. Now since there is no increment in the value of ‘i’ after this point, the condition 8 <= 9 will always hold true, and the loop continues indefinitely!

Let us write the indefinite loop version of this program and use the break statement to terminate it after 15 iterations.

Example 41: This program uses the break and continue statement to create an indefinite loop and an escape sequence.

In []: # This program is meant to print 1-10 and skip 8

       # It runs indefinitely until a break

  ​ i = 1        # loop variables

  ​ j = 1

  ​ while i<=10:

     ​ ​ print ('Iteration ',j)  

     ​ ​ if j >= 15:

         ​      break      # loop escape sequence

     ​ ​ j = j+1

   ​ ​ if i == 8:

         ​ ​ continue       # indefinite loop

     ​ ​ print('\tvalue =',i)    

     ​ ​ i = i+1  

      else: print('\nThe program has ended')

Out[]: Iteration  1

​ ​ value = 1

 Iteration  2

​ ​ value = 2

 Iteration  3

​ ​ value = 3

 Iteration  4

​ ​ value = 4

 Iteration  5

​ ​ value = 5

 Iteration  6

​ ​ value = 6

 Iteration  7

​ ​ value = 7

 Iteration  8

 Iteration  9

 Iteration  10

 Iteration  11

 Iteration  12

 Iteration  13

 Iteration  14

 Iteration  15

See how the loop became indefinite at output ‘7’ as explained earlier.

Tip: Always make sure your loop can end or insert a break statement to handle such errors.

For loop

These loops are used for iterating through a sequence of values. It executes a statement of code per each element present in the target sequence. Like while loops, they are also subject to an indefinite execution depending on how they are declared.

The for loop has the following syntax:

for iterator in sequence:

 ​ statement

Recall the ‘in’ operator from our comparison examples. It is a for loop operator.

The variable or name of iterator does not matter in the for loop, it could be anything; however, the sequence must be properly specified along with the loop statement.

Note: While it is not common for for loops to be infinite in Python, it is possible. Since the for loop iterates through a sequence, it is logical that the duration of the loop is dependent on the size of the sequence. If the sequence were infinite in length, then the for loop will run forever.

Below is a sample of an infinite for loop. Notice how the list will grow indefinitely through the increment of the list size and elements via the .append list method.

In []: ​ Indefinite_list = [0]

 ​ ​ for x in Indefinite_list:

    ​ ​     Indefinite_list.append(0)

    ​ ​     print(x)

The output is not included here because the code runs forever. You may try running this program, however, when the code runs indefinitely, click the ‘Kernel’ tab at the top of your Jupyter notebook and select ‘restart kernel and clear output’. This will restart that current cell and stop the loop immediately.

Now let us try some for loop examples.

Example 42: Let us write a code that prints all the elements in a list.

In []: # This program accepts a list and prints its elements numbers

  ​ List = eval (input ('Please enter a sequence of numbers using [ ] :\t'))

  ​ for value in List:

     ​ print ('\nValue',value,'is : ',List[value-1])

Out[]: Please enter a sequence of numbers using [ ] :              [1,2,3,4,5]

  ​ Value 1 is :  1

  ​ Value 2 is :  2

  ​ Value 3 is :  3

  ​ Value 4 is :  4

  ​ Value 5 is :  5

Recall, the for loop requires a sequence for its iteration to occur. In cases where you need a certain number of iterations and want to declare an arbitrary sequence of values with the desired length, you can use Python’s range() function.

The range function creates a sequence of numerical values starting from the lower arguments to a number of elements specified by the higher argument.

Syntax:​             range(x,y)

Here, x is the starting value of the sequence with y values, and the last value of a range is usually y -1.

Example 43: Create a list of 10 separate integers between 0 and 10.

In []: # using the range function

 ​ List = list(range(0,10))

 ​ print(List)

Out[]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Tip: In this example, the list was created using the list() method. However, the range function has been demonstrated.

Exercise: Write a for loop that iterates through a List of 10 elements, and prints the value of each element along with the iteration. 

Hint: use the range() function and list() method to generate your list. Also, try different variations of this exercise (more practical and complicated applications).

List Comprehension

This is a simple way of running for loops and some other conditional operations with lists in Python.

Syntax:

      [statement for iterator in sequence if condition]

      for iterator in sequence:     # Equivalent code

​ ​ ​    if condition:  

Example 44:   let us use List comprehension to square all the elements in a list

In []: trial_List = list(range(10))

      [(trial_List[item])**2 for item in trial_List]

Out[]: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

To use the IF statement part of the syntax, we consider a quick IF statement syntax allowed by Python.

                          An if condition else B

This allows you to write a one-line IF statement that executes the action/statement ‘A ’ if condition is met, else it evaluates B . Hence, for the IF statement part of List comprehension syntax, we can consider the preceding ‘ statement for iterator in sequenc e ’ as ‘A ’ which executes if a condition is met.

Example 45: Let us use the IF part of list comprehension to grab only the first three letters of each string item in a list.

In []: # Grab first three letters of strings

  ​ List = ['Matrix','Trilogy',1,3.4,'Cattle']

  ​ # grab loop

  ​ new_List = [things[:3] for things in List if type(things)== str]

  ​ print(new_List)

Out[]: ['Mat', 'Tri', 'Cat']

See how useful this code is and compact too. Here we iterate through the List using the iterator ‘things’, and we asked Python to grab the slice of ‘things’ (which would be the element it indexes at any instant in the loop) up till the third element i.e. things [:3] (recall the slice/range grabbing examples and exercises?). All these actions are then controlled by the last IF statement that checks if the element indexed by ‘things’ has a type == str i.e. is a string. Easy enough?

Exercise: To the list in example 45, add the following words ‘Jeep’, ‘Man’ and ‘Go’. Run the same code from example 45 with the updated list, however, only grab the first 3 letters of words that have at least 3 letters i.e. all items in your output list must have three letters.

Hint: Use the list .extend() method from previous examples to update the list. Also, use the function len() to check for the number of letters in a word. Finally, the IF statement can take logical operators ‘and, or, not’ .

With list comprehension, using the IF-ELSE statement is a little different. In this case, the IF-ELSE statement comes first, and then the for loop.

Syntax: [A if condition else B for iterator in sequence]

This is just like the one-lined IF statement, however with a for loop alongside it.

Example 46: Write a program that accepts user input for the number of list elements. Then the program creates a numeric list from 1 to that number, with values being squares of each odd number, and half for the even numbers. 

In []: # Create user defined list

nitems = eval(input('Please enter number of list items: '))

  ​ new_list  = [x**2 if x%2 != 0 else x/2 for x in range(1,nitems+1)]

  ​ print('\n  Here is your custom list\n\n',new_list)

Out[]: Please enter the number of list items: 10

  ​ Here is your custom list

  ​ [1, 1.0, 9, 2.0, 25, 3.0, 49, 4.0, 81, 5.0]

See how we created such a powerful program in 3 lines. That’s the advantage of list comprehension.

Tip: Take time to review the syntax for list comprehension, as it will be very useful in writing concise codes in the future.

There are other comprehension types in Python, like the dictionary comprehension; however, list comprehension is what we need for now.

Functions

In Python, or any other programming language, the use of functions cannot be underestimated. It allows the programmer to call certain code blocks to perform an action without having to write such code every time. Python has some in-built functions like the ‘range ()’, ‘list ()’, ‘type ()’, etc. that we have used in previous examples. As noticed, using these functions saved us time in their use. However, since Python is more like a tool/template for writing various programs that are supported within its library, it can hardly contain functions that would perform any action a programmer might require. This is why there is an option for defining custom functions in Python. With this option, you can write and call your own code for performing a larger operation.

To define a function in Python, we use the def () keyword. You may choose to interpret the keyword to mean ‘define’. It makes it easier to comprehend.

Syntax: ​ ​ def function_name(‘arguments’):

​     ​ ​    lines of code

The function_name is the whatever you wish to name your function as. This should be a name that makes sense as regards what the function does. Functions also accept arguments , which are inputs for the function to evaluate. Depending on the function you choose to define, the arguments can be of anything, or nothing.

Seems easy enough, right? Let us define some functions and call them to perform certain actions.

Example 47: Define a function that prints (‘Hello User’) when called.

In []:​ def greet(name):

    ​ ​     print('Hello ',name)

Now, let us call the function

In []:​ greet('David')

​ ​ Hello David

In []: ​ greet ('Python User guy')

​ ​ Hello  Python User guy

See, we only needed to call the function and pass in our argument. With this method, we can output a greeting to any string value without having to write any print statement.

It is good practice to include a ‘docstring’ when writing a function. These are comments (usually multi-line) that tell the user of the function how to call it and what it does. In Jupyter notebook, once the name of a function is typed, press shift key + tab in windows, and the docstring is shown. You can try this out for any of the functions we have used so far.

Figure 4: Docstring example for the print function.

To include a docstring for your function, just add a multi-line comment using the three quotes method. You can now check for the documentation of your function.

Let us try to recreate the example 46 using a function. The function prompts the user for the required list length, and then outputs a custom list. We would also include a documentation.

Example 48:

In []: ​ # function that creates a user-defined list

     ​ def custom_list(nitems):

     ​ """

    ​ This function accepts a numeric value and creates a list.

    ​ The list contains squares of odd numbers within the range

    ​ of 1 and user input, as well as, half values for the even

     ​ numbers in that range.

     ​ syntax: custom_list(numeric value)

     ​ """  

     ​ if nitems > 0:      # This ensures user compliance

     ​ new_list  = [x**2 if x%2 != 0 else x/2 for x in range(1,nitems+1)]

    ​ else: print('Enter a valid number !')  # in case of a fault.

     ​ print('\n Here is your custom list\n\n',new_list)  

Now, we have created our function. Let us call the function to view the action.

In []:​ # calling the function

          custom_list(5)

Out[]: ​ Here is your custom list

 ​ ​ [1, 1.0, 9, 2.0, 25]

The function works! However, since we only printed the output, we cannot access the list created by the function. Assuming we need to use that list for other operations within our code rather than just to view it, we can use the return statement to assign our function output to a variable, which we can then use later.

This is just a variation of the code in example 48.

Example 49:

In []: # function that creates a user-defined list and returns a value

      ​ def custom_list(nitems):

     ​ """

    ​ This function accepts a numeric value and returns a list.

    ​ The list contains squares of odd numbers within the range

    ​ of 1 and user input, as well as, half values for the even

     ​ numbers in that range.

     ​ syntax: List_name = custom_list(numeric value)

     ​ """  

     ​ if nitems > 0:      # This ensures user compliance

  ​ new_list  = [x**2 if x%2 != 0 else x/2 for x in range(1,nitems+1)]

    ​ else: print('Enter a valid number !')  # in case of a fault.

     ​ return new_list

Now let us call our function to assign the list to a variable ‘List’.

In []: ​ List = custom_list(10)

       ​ List

Out[]: [1, 1.0, 9, 2.0, 25, 3.0, 49, 4.0, 81, 5.0]

Notice how calling ‘List’ produces an output? The function works properly.

Exercise: Try creating a function that performs a variation of the action in example 45. The function accepts an input list (of mixed types), and extracts only numbers from the list.

Tip: for detecting numbers, you can use a variation of the method illustrated in example 45 i.e. type(item) == int or double. Another method is to use the .isdigit() method, i.e. item.isdigits  

Example 50: Illustrating the .isdigits() method.

In []: string = '123456ABC' ​ # mixed string with digits and characters

  ​ A = [x for x in string if x.isdigit()]

  ​ A

Out[]: ['1', '2', '3', '4', '5', '6']

To conclude this introductory chapter to Python programming, we would look at a few more functions that would be useful is the next section. These include: the map function, filter, and lambda expressions. Let us discuss the lambda expressions first, as they are mostly useful to map functions and filters as well.

Lambda expression

This is also called an anonymous function. Lambda expressions are used in instances where you would not want to write a full function. In this case, you only need to use the function temporarily, and defining an entire function for just that seems redundant.

It also reduces the effort in writing code for defining functions. It has the following syntax:                 var = lambda argument: statemen t

A function can equally be expressed in one line as such:

def function_name (argument): return output

Example 51: let’s write a quick function that squares a number and returns a value.

In []:​ def square_value (number): return number**2   #definition

      square_value(10)  # calling the function

Out []:​ 100

Alternatively, using lambda expression:

In []: ​ square = lambda number:number**2

      square(10)

Out[]:​ 100

The lambda expression in example 51, can be compared with the square_value function. It can be observed that some words have been eliminated: def, return and function_name. Note that lambda expressions have to be assigned to a variable, and then the variable acts as a function as illustrated by the ‘square’ variable which achieves the same result as the square_value function.

Map function

The map function is another convenient way of working with lists, especially in cases where there is a need to pass the elements of a list to a function in an iteration. As expected, for loops are the first consideration for such a task. However, the map function, which iterates through every element in a list and passes them to a function argument, simplifies such task.

Syntax: ​ map (function, sequence/list)

To return the map as a list of values, use the list () function and assign to a variable.

var =list (map (function, sequence/list))

You could pass an actual function to map, or more practically, use a lambda expression.

Example 52: Let us write a code using the map function to square all the elements in a list.

In []: # This program squares all the elements in a list

    ​ List = list(range(1,11))

  ​ # using the lambda expression 'square'

    ​ new_list = list(map(square,List))

    ​ new_list

Out[]: [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Exercise: Using the lambda expression and map function, write a program that creates a list of values and displays all the elements in the list.

Filter function

These are used for extracting specific elements of a sequence based on a Boolean function. The Boolean function can also be created using a lambda expression, and the filter returns the expected values that meet the filter criteria (consider it as a simplified version of an if statement).

Syntax: ​ filter (function, sequence/list)

To return the map as a list of values, use the list () function and assign to a variable.

var = list (filter (function, sequence/list))

Example 53: Extract only the even numbers from the new_list created from example 52 using the filter function.

In []: # Extracting even numbers

      ​ criteria = lambda value:value%2==0   # even number identifier

     # using the new_list from example 52

even_list = list(filter(criteria,new_list))

     even_list

Out[]:     [4, 16, 36, 64, 100]

The lambda, map and filter are usually used respectively. It is important to master them and it will considerably improve your skills with list manipulation which is very important in the subsequent section.

This concludes the introductory part to Python and all the pre-requisites for following the lessons in the next chapter. Complete the following exercises to test your current knowledge of major lessons from this chapter. Best of luck!

Exercises:

These exercises test your skills on all you have learned in this chapter. Try to solve them yourself, and if you find any difficulty, review the examples and syntax all over.

Given the following variables: quantity = ‘Gravity’, unit = ‘m/s^2’, and value = 10, use the .format() method to print the following statement: Gravity has a value of 10 m/s^2.

Write a function that prints True if the word ‘good ’ occurs in any list of strings.

Use the lambda expression and filter function to extract words that do not start with the letter ‘b ’ in the following list [‘bread’,’rice’,’butter’,’beans’,’pizza’,’lasagna’,’eggs’]

Given this nested list, use indexing to grab the word "hello Python”.

list = [1,2,[3,4],[5,[100,200,['hello Python']],23,11],1,7]

Given this nested dictionary grab the word "hello Python"

d = {'k1':[1,2,3,{'tricky':['oh','man','inception',{'target':[1,2,3,'hello']}]}]}

Write a function that accepts two arguments. The first argument is the student name, the other is the student score. Let the program print the student name and grade.

The output is dependent on the following criteria:

Criteria:

If the student score is less than 40: ​ Hello student name you had an F

If the student score is between 40 and 44: ​ Hello student name you had an E

If the student score is between 45 and 49: ​ Hello student name you had a D

If the student score is between 50 and 59: ​ Hello student name you had a C

If the student score is between 60 and 69: ​ Hello student name you had a B

If the student score is from 70 and above: ​ Hello student name you had an A, Cool!

Chapter 2

Data Analysis with Python

Data analysis includes all the techniques and processes used in extracting information from raw data. Since raw data is usually unstructured in form, and hardly informative, the need to organize such data becomes even more important. While there are many extra tools that can be used for handling data analysis i.e. Microsoft Excel, R-language, SQL, etc., most data scientist prefer to use Python due to its extensive libraries and support packages for data analysis. The most popular packages/frameworks which we would be exploring in this chapter are NumPy and Pandas.

NumPy

This is the numerical Python package which supports vector and matrix operations. It is a very popular Python package for scientific, mathematical and engineering programming; especially for linear algebraic problems. To a large extent, numeric data can be simplified into arrays (vectors or matrices, depending on dimensions), and this is why NumPy is equally useful in data manipulation and organization.

Package Installation

To get started with NumPy, we have to install the package into our version of Python. While the basic method for installing packages to Python is the pip instal l method, we will be using the conda instal l method. This is the recommended way of managing all Python packages and virtual environments using the anaconda framework.

Since we installed a recent version of Anaconda, most of the packages we need would have been included in the distribution. To verify if any package is installed, you can use the conda lis t command via the anaconda prompt. This displays all the packages currently installed and accessible via anaconda. If your intended package is not available, then you can install via this method:

First, ensure you have an internet connection. This is required to download the target package via conda. Open the anaconda prompt, then enter the following code:

Conda install package

Note : In the code above, ‘package’ is what needs to be installed e.g. NumPy, Pandas, etc.

As described earlier, we would be working with NumPy arrays. In programming, an array is an ordered collection of similar items. Sounds familiar? Yeah, they are just like Python lists, but with superpowers. NumPy arrays are in two forms: Vectors, and Matrices. They are mostly the same, only that vectors are one-dimensional arrays (either a column or a row of ordered items), while a matrix is 2-dimensional (rows and columns). These are the fundamental blocks of most operations we would be doing with NumPy. While arrays incorporate most of the operations possible with Python lists, we would be introducing some newer methods for creating, and manipulating them.

To begin using the NumPy methods, we have to first import the package into our current workspace. This can be achieved in two ways:

import numpy as np

Or

from numpy import *

In Jupyter notebook, enter either of the codes above to import the NumPy package. The first method of import is recommended, especially for beginners, as it helps to keep track of the specific package a called function/method is from. This is due to the variable assignment e.g. ‘np’, which refers to the imported package throughout the coding session.

Notice the use of an asterisk in the second import method. This signifies ‘everything/all’ in programming. Hence, the code reads ‘from NumPy import everything!! ’

Tip: In Python, we would be required to reference the package we are operating with e.g. NumPy, Pandas, etc. It is easier to assign them variable names that can be used in further operations. This is significantly useful in a case where there are multiple packages being used, and the use of standard variable names such as: ‘np’ for NumPy, ‘pd’ for Pandas, etc. makes the code more readable.

Example 55:    Creating vectors and matrices from Python lists.

Let us declare a Python list.

In []: ​ # This is a list of integers

 ​ ​ Int_list = [1,2,3,4,5]

      ​ Int_list

Out[]: ​ [1,2,3,4,5]

Importing the NumPy package and creating an array of integers.

In []: ​ # import syntax

​ ​ import numpy as np

​ ​ np.array(Int_list)

Out[]: ​ array([1, 2, 3, 4, 5])

Notice the difference in the outputs? The second output indicates that we have created an array, and we can easily assign this array to a variable for future reference.

To confirm, we can check for the type.

In []: ​ x = np.array(Int_list)

       ​ type(x)

Out[]: ​ numpy.ndarray

We have created a vector, because it has one dimension (1 row). To check this, the ‘ndim’ method can be used.

In []: ​ x.ndim    # this shows how many dimensions the array has

Out[]: ​ 1

Alternatively, the shape method can be used to see the arrangements.

In []: ​ x.shape   # this shows the shape

Out[]: ​ (5,)

Python describes matrices as (rows, columns ) . In this case, it describes a vector as (number of elements, ) .

To create a matrix from a Python list, we need to pass a nested list containing the elements we need. Remember, matrices are rectangular, and so each list in the nested list must have the same size.

In []: # This is a matrix

  ​ x = [1,2,3]

  ​ y = [4,5,6]

  ​ my_list = [y,x]  # nested list

  ​ my_matrix = np.array(my_list)  # creating the matrix

  ​ A = my_matrix.ndim

  ​ B = my_matrix.shape

  ​ # Printing

  ​ print('Resulting matrix:\n\n',my_matrix,'\n\nDimensions:',A,

  ​ '\nshape (rows,columns):',B)

Out[]: Resulting matrix:

  ​ [[4 5 6]

  ​ [1 2 3]]

  ​ Dimensions: 2

  ​ shape (rows,columns): (2, 3)

Now, we have created a 2 by 3 matrix. Notice how the shape method displays the rows and columns of the matrix. To find the transpose of this matrix i.e. change the rows to columns, use the transpose ( ) method.

In []: ​ # this finds the transpose of the matrix

​ ​ t_matrix = my_matrix.transpose()

         ​ t_matrix

Out[]: ​ array([[4, 1],

       ​ ​ [5, 2],

       ​ ​ [6, 3]])

Tip: Another way of knowing the number of dimensions of an array is by counting the square-brackets that opens and closes the array (immediately after the parenthesis). In the vector example, notice that the array was enclosed in single square brackets. In the two-dimensional array example, however, there are two brackets. Also, tuples can be used in place of lists for creating arrays.

There are other methods of creating arrays in Python, and they may be more intuitive than using lists in some application. One quick method uses the arange( ) function.

Syntax: np.arange(start value, stop value, step size, dtype = ‘type’)

This method is similar to the range( ) method we used in example 43. In this case, we do not need to pass its output to the list function, our result is an array object of a data type specified by ‘dtype’.

Example 56 : Creating arrays with the arange() function.

We will create an array of numbers from 0 to 10, with an increment of 2 (even numbers).

In []: ​ # Array of even numbers between 0 and 10

​ ​ Even_array = np.arange(0,11,2)

​ ​ Even_array

Out[]: ​ array([ 0,  2,  4,  6,  8, 10])

Notice it behaves like the range () method form our list examples. It returned all even values between 0 and 11 (10 being the maximum). Here, we did not specify the types of the elements.

Tip: Recall, the range method returns value up to the ‘stop value – 1’; hence, even if we change the 11 to 12, we would still get 10 as the maximum.

Since the elements are numeric, they can either be integers or floats. Integers are the default, however, to return the values as floats, we can also specify the numeric type.

In []: ​ Even_array2 = np.arange(0,11,2, dtype='float')

​ ​ Even_array2

Out[]: ​ array([ 0.,  2.,  4.,  6.,  8., 10.])

Another handy function for creating arrays is linspace( ) . This returns a numeric array of linearly space values within an interval. It also allows for the specification of the required number of points, and it has the following syntax:

np.linspace(start value, end value, number of points)

At default, linspace returns an array of 50 evenly spaced points within the defined interval.

Example 57 : Creating arrays of evenly spaced points with linspace()

In []: # Arrays of linearly spaced points

​ A = np.linspace(0,5,5) # 5 equal points between 0 & 5

​ B = np.linspace (51,100) # 50 equal points between 51 & 100

       ​ print ('Here are the arrays:\n')

​ A

​ B

Here are the arrays:

Out[ ]: array([0.  , 1.25, 2.5 , 3.75, 5.  ])

Out[ ]: array([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38., 39., 40., 41., 42., 43., 44., 45., 46., 47., 48., 49., 50.])

Notice how the second use of linspace did not require a third argument. This is because we wanted 50 equally spaced values, which is the default. The ‘dtype’ can also be specified like we did with the range function.

Tip 1: Linspace arrays are particularly useful in plots. They can be used to create a time axis or any other required axis for producing well defined and scaled graphs.

Tip 2: The output format in the example above is not the default way for output in Jupyter notebook. Jupyter displays the last result per cell, at default. To display multiple results (without having to use the print statement every-time), the output method can be changed using the following code.

In[]: # Allowing Jupyter output all results per cell.

​ # run the following code in a Jupyter cell.

  ​ from IPython.core.interactiveshell import InteractiveShell

  ​ InteractiveShell.ast_node_interactivity = "all"

There are times when a programmer needs unique arrays like the identity matrix, or a matrix of ones/zeros. NumPy provides a convenient way of creating these with the zeros( ) , ones( ) and eye( ) functions.

Example 58: creating arrays with unique elements.

Let us use the zeros () function to create a vector and a matrix.

In []: np.zeros(3)  # A vector of 3 elements

  ​ np.zeros((2,3)) # A matrix of 6 elements i.e. 2 rows, 3 columns

Out[]: array([0., 0., 0.])

Out[]: array([[0., 0., 0.],

     ​   [0., 0., 0.]])

Notice how the second output is a two-dimensional array i.e. two square brackets (a matrix of 2 columns and 3 rows as specified in the code).

The same thing goes for creating a vector or matrix with all elements having a value of ‘1’.

In []: np.ones(3)  # A vector of 3 elements

  ​ np.ones((2,3)) # A matrix of 6 elements i.e. 2 rows, 3 columns

Out[]: array([1., 1., 1.])

Out[]: array([[1., 1., 1.],

     ​    [1., 1., 1.]])

Also, notice how the code for creating the matrices requires the row and column instructions to be passed as a tuple. This is because the function accepts one input, so multiple inputs would need to be passed as tuples or lists in the required order (Tuples are recommended. Recall, they are faster to operate.).

In the case of the identity matrix, the function eye () only requires one value. Since identity matrices are always square, the value passed determines the number of rows and columns.

In []: np.eye(2)  # A matrix of 4 elements 2 rows, 2 columns

  ​ np.eye(3)  # 3 rows, 3 columns

Out[]: array([[1., 0.],

        ​      [0., 1.]])

Out[]: array([[1., 0., 0.],

        ​      [0., 1., 0.],

        ​      [0., 0., 1.]])

NumPy also features random number generators. These can be used for creating arrays, as well as single values, depending on the required application. To access the random number generator, we call the library via np.rando m , and then choose the random method we prefer. We will consider three methods for generating random numbers: rand( ) , randn( ) , and randint( ) .

Example 59: Generating arrays with random values.

Let us start with the rand () method. This generates random, uniformly distributed numbers between 0 and 1.

In []: np.random.rand (2)    # A vector of 2 random values

  ​ np.random.rand (2,3)  # A matrix of 6 random values

Out[]: array([0.01562571, 0.54649508])

Out[]: array([[0.22445055, 0.35909056, 0.53403529],

        ​   [0.70449515, 0.96560456, 0.79583743]])

Notice how each value within the arrays are between 0 & 1. You can try this on your own and observe the returned values. Since it is a random generation, these values may be different from yours. Also, in the case of the random number generators, the matrix specifications are not required to be passed as lists or tuples, as observed in the second line of code.

The randn () method generates random numbers from the standard normal or Gaussian distribution. You might want to brush up on some basics in statistics, however, this just implies that the values returned would have a tendency towards the mean (which is zero in this case) i.e. the values would be centered around zero.

In []: np.random.randn (2)     # A vector of 2 random values

  ​ np.random.randn (2,3)  # A matrix of 6 random values

Out[]: array([ 0.73197866, -0.31538023])

Out[]: array([[-0.79848228, -0.7176693 , 0.74770505],

        ​    [-2.10234448,  0.10995745, -0.54636425]])

The randint() method generates random integers within a specified range or interval. Note that the higher range value is exclusive (i.e. has no chance of being randomly selected), while the lower value is inclusive (could be included in the random selection).

Syntax: np.random(lower value, higher value, number of values, dtype)

If the number of values is not specified, Python just returns a single value within the defined range.

In []: np.random.randint (1,5)         # A random value between 1 and 5

  ​ np.random.randint (1,100,6)     # A vector of 6 random values

​ np.random.randint (1,100,(2,3)) # A matrix of 6 random values

Out[]: 4

Out[]: array([74, 42, 92, 10, 76, 43])

Out[]: array([[92,  9, 99],

             [73, 36, 93]])

Tip: Notice how the size parameter for the third line was specified using a tuple. This is how to create a matrix of random integers using randint.

Example 59 : Illustrating randint().

Let us create a fun dice roll program using the randint() method. We would allow two dice, and the function will return an output based on the random values generated in the roll.

In []: # creating a dice roll game with randint()

  ​ # Defining the function

  ​ def roll_dice():

     ​      """ This function displays a

     ​      dice roll value when called"""

    ​ dice1 = np.random.randint(1,7) # This allows 6 to be inclusive

    ​      dice2 = np.random.randint(1,7)

​ # Display Condition.

     ​      if dice1 == dice2:

       ​ print('Roll: ',dice1,'&',dice2,'\ndoubles !')

         ​    if dice1 == 1:

          ​        print('snake eyes!\n')  

     ​      else:

        ​ print('Roll: ',dice1,'&',dice2)

In []: ​ # Calling the function

  ​ roll_dice()

Out[]: ​ Roll:  1 & 1

  ​ doubles !

  ​ snake eyes!

Hint:  Think of a fun and useful program to illustrate the use of these random number generators, and writing such programs will improve your chances of comprehension. Also, a quick review of statistics, especially measures of central tendency & dispersion/spread will be useful in your data science journey.

Manipulating arrays

Now that we have learned how to declare arrays, we would be proceeding with some methods for modifying these arrays. First, we will consider the reshape ( ) method, which is used for changing the dimensions of an array.

Example 60 : Using the reshape() method.

Let us declare a few arrays and call the reshape method to change their dimensions.

In []: freq = np.arange(10);values = np.random.randn(10)

       freq; values

Out[]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

Out[]: array([ 1.33534821,  1.73863505,  0.1982571 , -0.47513784, 1.80118596, -1.73710743,
​ -0.24994721, 1.41695744, -0.28384007,  0.58446065])

Using the reshape method, we would make ‘freq’ and ‘values’ 2 dimensional.

In []: np.reshape(freq,(5,2))

Out[]: array([[0, 1],

        ​ [2, 3],

        ​ [4, 5],

        ​ [6, 7],

        ​ [8, 9]])

In []: np.reshape(values,(2,5))

Out[]: array([[ 1.33534821,  1.73863505,  0.1982571 , -0.47513784,  1.80118596],

  ​ [-1.73710743, -0.24994721,  1.41695744,      -0.28384007,  0.58446065]])

Even though the values array still looks similar after reshaping, notice the two square brackets that indicate it has been changed to a matrix. The reshape method comes in handy when we need to do array operations, and our arrays are inconsistent in dimensions. It is also important to ensure the new size parameter passed to the reshape method does not differ from the number of elements in the original array. The idea is simple: when calling the reshape method, the product of the size parameters must equal the number of elements in the original array.

As seen in Example 60, the size parameter passed as a tuple to the reshape methods gives a value of 10 when multiplied, and this is also the number of elements in ‘freq’ and ‘values’ respectively.

There are times when we may need to find the maximum and minimum values within an array (or real-world data), and possibly the index of such maximum or minimum values. To get this information, we can use the .max( ) , .min( ) , .argmax( ) and .argmin( ) methods respectively.

Example 61:

Let us find the maximum and minimum values in the ‘values’ array, along with the index of the minimum and maximum within the array.

In []: A = values.max();B = values.min();

       ​ C = values.argmax()+1; D = values.argmin()+1

      print('Maximum value: {}\nMinimum Value: {}\

      \nItem {} is the maximum value, while item {}\

  ​ is the minimum value'.format(A,B,C,D))

Output

Maximum value: 1.8011859577930067

Minimum Value: -1.7371074259180737

Item 5 is the maximum value, while item 6 is the minimum value

A few things to note in the code above: The variables C&D, which defines the position of the maximum and minimum values are evaluated as shown [by adding 1 to the index of the maximum and minimum values obtained via argmax ( ) and argmin ( ) ], because Python indexing starts at zero. Python would index maximum value at 4, and minimum at 5, which is not the actual positions of these elements within the array (you are less likely to start counting elements in a list from zero! Unless you are Python, of course.).

Another observation can be made in the code. The print statement is broken across a few lines using enter. To allow Python to know that the next line of code is a continuation, the backslash ‘\’ is used. Another way would be to use three quotes for a multiline string.

Indexing and selecting arrays  

Array indexing is very much similar to List indexing with the same techniques of item selection and slicing (using square brackets). The methods are even more similar when the array is a vector.

Example 62:

In []: # Indexing a vector array (values)

  ​ values

  ​ values[0]   # grabbing 1st item

  ​ values[-1]  # grabbing last item

  ​ values[1:3] # grabbing 2nd & 3rd item

  ​ values[3:8] # item 4 to 8

Out[]: array([ 1.33534821,  1.73863505,  0.1982571 , -0.47513784,  1.80118596,  -1.73710743,       -0.24994721,  1.41695744, -0.28384007,  0.58446065])

Out[]: 1.3353482110285562

Out[]: 0.5844606470172699

Out[]: array([1.73863505, 0.1982571 ])

Out[]: array([-0.47513784,  1.80118596, -1.73710743, -0.24994721,  1.41695744])

The main difference between arrays and lists is in the broadcast property of arrays. When a slice of a list is assigned to another variable, any changes on that new variable does not affect the original list. This is seen in the example below:

In []: num_list = list(range(11))  # list from 0-10

  ​ num_list                     # display list

  ​ list_slice = num_list[:4]   # first 4 items

  ​ list_slice                   # display slice

  ​ list_slice[:] = [5,7,9,3]    # Re-assigning elements   

  ​ list_slice                  # display updated values

  ​ # checking for changes

  ​ print(' The list changed !') if list_slice == num_list[:4]\

  ​ else print(' no changes in original list')

Out[]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Out[]: [0, 1, 2, 3]

Out[]: [5, 7, 9, 3]

  ​ no changes in the original list

For arrays, however, a change in the slice of an array also updates or broadcasts to the original array, thereby changing its values.

In []: # Checking the broadcast feature of arrays

  ​ num_array = np.arange(11)     # array from 0-10

  ​ num_array                     # display array

  ​ array_slice = num_array[:4]  # first 4 items

  ​ array_slice                   # display slice

  ​ array_slice[:] = [5,7,9,3]    # Re-assigning elements   

  ​ array_slice                   # display updated values

  ​ num_array

Out[]: array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10])

Out[]: array([0, 1, 2, 3])

Out[]: array([5, 7, 9, 3])

Out[]: array([ 5,  7,  9,  3,  4,  5,  6,  7,  8,  9, 10])

This happens because Python tries to save memory allocation by allowing slices of an array to be like shortcuts or links to the actual array. This way it doesn’t have to allocate a separate memory location to it. This is especially ingenious in the case of large arrays whose slices can also take up significant memory. However, to take up a slice of an array without broadcast, you can create a ‘slice of a copy’ of the array. The array.copy( ) method is called to create a copy of the original array.

In []: ​ # Here is an array allocation without broadcast

  ​ num_array     # Array from the last example

       ​ # copies the first 4 items of the array copy

       ​ array_slice = num_array.copy()[:4]

       ​ array_slice                   # display array

      ​ array_slice[:] = 10           # re-assign array

       ​ array_slice                   # display updated values

       ​ num_array                    # checking original list

Out[]: ​ array([ 5,  7,  9,  3,  4,  5,  6,  7,  8,  9, 10])

Out[]: ​ array([5, 7, 9, 3])

Out[]: ​ array([10, 10, 10, 10])

Out[]: ​ array([ 5,  7,  9,  3,  4,  5,  6,  7,  8,  9, 10])

Notice that the original array remains unchanged.

For two-dimensional arrays or matrices, the same indexing and slicing methods work. However, it is always easy to consider the first dimension as the rows and the other as the columns. To select any item or slice of items, the index of the rows and columns are specified. Let us illustrate this with a few examples:

Example 63 : Grabbing elements from a matrix

There are two methods for grabbing elements from a matrix: array_name[row][col] or array_name[row,col ] .

In []: # Creating the matrix

  ​ matrix = np.array(([5,10,15],[20,25,30],[35,40,45]))

  ​ matrix       #display matrix

  ​ matrix[1]    # Grabbing second row

  ​ matrix[2][0] # Grabbing 35

  ​ matrix[0:2]  # Grabbing first 2 rows

  ​ matrix[2,2]   # Grabbing 45

Out[]: array([[ 5, 10, 15],

        ​   [20, 25, 30],

        ​   [35, 40, 45]])

Out[]: array([20, 25, 30])

Out[]: 35

Out[]: array([[ 5, 10, 15],

             [20, 25, 30]])

Out[]: 45

Tip: It is recommended to use the array_name[row,col ] method, as it saves typing and is more compact. This will be the convention for the rest of this section.

To grab columns, we specify a slice of the row and column. Let us try to grab the second column in the matrix and assign it to a variable column_slice.

In []: # Grabbing the second column

  ​ column_slice = matrix[:,1:2] # Assigning to variable

  ​ column_slice

Out[]: array([[10],

        ​   [25],

        ​   [40]])

Let us consider what happened here. To grab the column slice, we first specify the row before the comma. Since our column contains elements in all rows, we need all the rows to be included in our selection, hence the ‘ : ’ sign for all. Alternatively, we could use ‘0 : ’ , which might be easier to understand. After selecting the row, we then choose the column by specifying a slice from ‘1:2 ’ , which tells Python to grab from the second item up to (but not including) the third item. Remember, Python indexing starts from zero.

Exercise: Try to create a larger array, and use these indexing techniques to grab certain elements from the array. For example, here is a larger array:

In []: # 5 10 Array of even numbers between 0 and 100.

large_array = np.arange(0,100,2).reshape(5,10)

large_array     # show

Out[]: array([[ 0,  2,  4,  6,  8, 10, 12, 14, 16, 18],

        ​   [20, 22, 24, 26, 28, 30, 32, 34, 36, 38],

        ​   [40, 42, 44, 46, 48, 50, 52, 54, 56, 58],

        ​   [60, 62, 64, 66, 68, 70, 72, 74, 76, 78],

        ​   [80, 82, 84, 86, 88, 90, 92, 94, 96, 98]])

Tip: Try grabbing single elements and rows from random arrays you create. After getting very familiar with this, try selecting columns. The point is to try as many combinations as possible to get you familiar with the approach. If the slicing and indexing notations are confusing, try to revisit the section under list or string slicing and indexing.

Click this link to revisit the examples on slicing: List indexing

Conditional selection

Consider a case where we need to extract certain values from an array that meet a Boolean criterion. NumPy offers a convenient way of doing this without having to use loops.

Example 64: Using conditional selection

Consider this array of odd numbers between 0 and 20. Assuming we need to grab elements above 11. We first have to create the conditional array that selects this:

In []: odd_array = np.arange(1,20,2)   # Vector of odd numbers

  ​ odd_array                        # Show vector

  ​ bool_array = odd_array > 11      # Boolean conditional array

  ​ bool_array

Out[]: array([ 1,  3,  5,  7,  9, 11, 13, 15, 17, 19])

Out[]: array([False, False, False, False, False, False,  True, True, True, True])

Notice how the bool_array evaluates to True at all instances where the elements of the odd_array meet the Boolean criterion.

The Boolean array itself is not usually so useful. To return the values that we need, we will pass the Boolean_array into the original array to get our results.

In []: useful_Array = odd_array[bool_array]  # The values we want

  ​ useful_Array

Out[]: array([13, 15, 17, 19])

Now, that is how to grab elements using conditional selection. There is however a more compact way of doing this. It is the same idea, but it reduces typing.

Instead of first declaring a Boolean_array to hold our truth values, we just pass the condition into the array itself, like we did for useful_array.

In []: # This code is more compact

  ​ compact = odd_array[odd_array>11] # One line

  ​ compact

Out[]: array([13, 15, 17, 19])

See how we achieved the same result with just two lines? It is recommended to use this second method, as it saves coding time and resources. The first method helps explain how it all works. However, we would be using the second method for all other instances in this book.

Exercise: The conditional selection works on all arrays (vectors and matrices alike). Create a two 3 3 array of elements greater than 80 from the ‘large_array’ given in the last exercise.

Hint: use the reshape method to convert the resulting array into a 3 3 matrix.

NumPy Array Operations

Finally, we will be exploring basic arithmetical operations with NumPy arrays. These operations are not unlike that of integer or float Python lists.

Array – Array Operations

In NumPy, arrays can operate with and on each other using various arithmetic operators. Things like the addition of two arrays, division, etc.

Example 65:

In []: # Array - Array Operations

  ​ # Declaring two arrays of 10 elements

  ​ Array1 = np.arange(10).reshape(2,5)

  ​ Array2 = np.random.randn(10).reshape(2,5)

  ​ Array1;Array2               # Show the arrays

  ​ # Addition

  ​ Array_sum = Array1 + Array2

  ​ Array_sum                  # show result array

  ​ #Subtraction

  ​ Array_minus = Array1 - Array2

  ​ Array_minus                 # Show array

  ​ # Multiplication

  ​ Array_product = Array1 * Array2

  ​ Array_product               # Show

  ​ # Division

  ​ Array_divide = Array1 / Array2

  ​ Array_divide                 # Show

Out[]: array([[0, 1, 2, 3, 4],

        ​   [5, 6, 7, 8, 9]])

array([[ 2.09122638,  0.45323217, -0.50086442,  1.00633093,  1.24838264],  [ 1.64954711,     -0.93396737,  1.05965475,  0.78422255,        -1.84595505]])

array([[2.09122638, 1.45323217, 1.49913558, 4.00633093, 5.24838264], [6.64954711, 5.06603263, 8.05965475, 8.78422255, 7.15404495]])

array([[-2.09122638,  0.54676783,  2.50086442,  1.99366907,  2.75161736], [ 3.35045289,  6.93396737,  5.94034525,  7.21577745, 10.84595505]])

array([[  0.      ,   0.45323217,  -1.00172885,   3.01899278, 4.99353055], [  8.24773555,      -5.60380425,   7.41758328,   6.27378038,     -16.61359546]])

array([[ 0.      ,  2.20637474, -3.99309655,  2.9811267 ,  3.20414581],  [ 3.03113501,     -6.42420727,  6.60592516, 10.20118591,       -4.875525  ]])

Each of the arithmetic operations performed are element-wise. The division operations require extra care however. In Python, most arithmetic errors in code throw a run-time error, which helps in debugging. For NumPy, however, the code could run with a warning issued.

Array – Scalar operations

Also, NumPy supports scalar with Array operations. A scalar in this context is just a single numeric value of either integer or float type. The scalar – Array operations are also element-wise, by virtue of the broadcast feature of NumPy arrays.

Example 66:

In []: #Scalar- Array Operations

  ​ new_array = np.arange(0,11)     # Array of values from 0-10

  ​ print('New_array')

  ​ new_array                       # Show

  ​ Sc = 100                        # Scalar value

  ​ # let us make an array with a range from 100 - 110 (using +)

  ​ add_array = new_array + Sc      # Adding 100 to every item

  ​ print('\nAdd_array')

  ​ add_array                       # Show

  ​ # Let us make an array of 100s (using -)

  ​ centurion = add_array - new_array

  ​ print('\nCenturion')

  ​ centurion                       # Show

  ​ # Let us do some multiplication (using *)

  ​ multiplex = new_array * 100    

  ​ print('\nMultiplex')

  ​ multiplex                       # Show

  ​ # division [take care], let us deliberately generate

  ​ # an error. We will do a divide by Zero.

  ​ err_vec = new_array / new_array

  ​ print('\nError_vec')

  ​ err_vec                        # Show

  ​ New_array

Out[]: array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10])

  ​ Add_array

Out[]: array([100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110])

  ​ Centurion

Out[]: array([100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100])

  ​ Multiplex

Out[]: array([   0,  100,  200,  300,  400,  500,  600,  700,  800,  900, 1000])

  ​ Error_vec

      ​ C:\Users\Oguntuase\Anaconda3\lib\site-packages\ipykernel_launcher.py:27:                RuntimeWarning: invalid value encountered in true_divide

array([nan,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.])

Notice the runtime error generated? This divide by zero value was caused by the division of the first element of new_arra y by itself, i.e. 0/ 0 . This would give a divide by zero error in normal Python environment and would not run the code. NumPy, however, ran the code and indicated the divide by zero in the Error_ve c array as a ‘nan ’ type (not-a-number). This also goes for values that evaluate to infinity, which would be represented by the value ‘+/- inf ’ (try 1/ 0 using NumPy array-scalar or array-array operation.).

Tip: Always take caution when using division to avoid such runtime errors that could later bug your code.

Universal Array functions

These are some built-in functions designed to operate in an element-wise fashion on NumPy arrays. They include mathematical, comparison, trigonometric, Boolean, etc. operations. They are called using the np.function_name(array) method.

Example 67 : A few Universal Array functions (U-Func)

In []: # Using U-Funcs

  ​ U_add = np.add(new_array,Sc)  # addition

  ​ U_add                          # Show

  ​ U_sub = np.subtract(add_array,new_array)

  ​ U_sub                          # Show

  ​ U_log = np.log(new_array)      # Natural log

  ​ U_log                         # Show

  ​ sinusoid = np.sin(new_array)  # Sine wave

  ​ sinusoid                       # Show

  ​ # Alternatively, we can use the .method

  ​ new_array.max()                # find maximum

  ​ np.max(new_array)              # same thing

Out[]: array([100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110])

Out[]: array([100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100])

C:\Users\Oguntuase\Anaconda3\lib\site-packages\ipykernel_launcher.py:8:     RuntimeWarning: divide by zero encountered in log

Out[]: array([      -inf, 0.        , 0.69314718, 1.09861229, 1.38629436, 1.60943791, 1.79175947, 1.94591015, 2.07944154, 2.19722458, 2.30258509])

Out[]: array([ 0.        ,  0.84147098,  0.90929743,  0.14112001, -0.7568025 , -0.95892427,         -0.2794155 ,  0.6569866 ,  0.98935825,  0.41211849, -0.54402111])

Out[]: 10

Out[]: 10

There are still many more functions available, and a full reference can be found in the NumPy documentation for Universal functions here: https://docs.scipy.org/doc/numpy/reference/ufuncs.html

Now that we have explored NumPy for creating arrays, we would consider the Pandas framework for manipulating these arrays and organizing them into data frames.

Pandas

This is an open source library that extends the capabilities of NumPy. It supports data cleaning and preparation, with fast analysis capabilities. It is more like Microsoft excel framework, but with Python. Unlike NumPy, it has its own built-in visualization features and can work with data from a variety of sources. It is one of the most versatile packages for data science with Python, and we will be exploring how to use it effectively.

To use pandas, make sure it is currently part of your installed packages by verifying with the conda lis t method. If it is not installed, then you can install it using the conda install panda s command; you need an internet connection for this.

Now that Pandas is available on your PC, you can start working with the package. First, we start with the Pandas series.

Series

This is an extension of the NumPy array. It has a lot of similarities, but with a difference in indexing capacity. NumPy arrays are only indexed via number notations corresponding to the desired rows and columns to be accessed. For Pandas series, the axes have labels that can be used for indexing their elements. Also, while NumPy arrays -- like Python lists, are essentially used for holding numeric data, Pandas series are used for holding any form of Python data/object.

Example 68 : Let us illustrate how to create and use the Pandas series

First, we have to import the Pandas package into our workspace. We will use the variable name pd for Pandas, just as we used np for NumPy in the previous section.

In []: import numpy as np   #importing numpy for use

       ​ import pandas as pd  # importing the Pandas package

We also imported the numpy package because this example involves a numpy array.

In []: # python objects for use

  ​ labels = ['First','Second','Third']   
                               # string list

  ​ values = [10,20,30]             # numeric list

  ​ array = np.arange(10,31,10)     # numpy array

  ​ dico = {'First':10,'Second':20,'Third':30}
                          # Python dictionary

  ​ # create various series

  ​ c = pd.Series(values)

  ​ print('Default series')

  ​ A                                      #show

  ​ B = pd.Series(values,labels)

  ​ print('\nPython numeric list and label')

  ​ B                                      #show

  ​ C = pd.Series(array,labels)

  ​ print('\nUsing python arrays and labels')

  ​ C                                      #show

  ​ D = pd.Series(dico)

  ​ print('\nPassing a dictionary')

  ​ D                                      #show

  ​ Default series

Out[]: 0    10

   ​ 1    20

  ​ 2    30

  ​ dtype: int64

  ​ Python numeric list and label

Out[]: ​ First     10

  ​ Second    20

  ​ Third     30

  ​ dtype: int64

  ​ Using python arrays and labels

Out[]: ​ First     10

  ​ Second    20

  ​ Third     30

  ​ dtype: int32

  ​ Passing a dictionary

Out[]: First     10

  ​ Second    20

  ​ Third     30

  ​ dtype: int64

We have just explored a few ways of creating a Pandas series using a numpy array, Python list, and dictionary. Notice how the labels correspond to the values? Also, the dtypes are different. Since the data is numeric and of type integer, Python assigns the appropriate type of integer memory to the data. Creating series from NumPy arrays returns the smallest integer size (int 32). The difference between 32 bits and 64 bits unsigned integers is the corresponding memory allocation. 32 bits obviously requires less memory (4bytes, since 8bits make a byte), and 64 bits would require double (8 bytes). However, 32bits integers are processed faster, but have a limited capacity in holding values, as compared with 64 bits.

Pandas series also support the assignment of any data type or object as its data points.

In []: pd.Series(labels,values)

Out[]: 10     First

      20    Second

      30     Third

      dtype: object

Here, the string elements of the label list are now the data points. Also, notice that the dtype is not ‘object’.

This kind of versatility in item operation and storage is what makes pandas series very robust. Pandas series are indexed using labels. This is illustrated in the following examples:

Example 69:

In []: # series of WWII countries

  ​ pool1 = pd.Series([1,2,3,4],['USA','Britain','France','Germany'])

  ​ pool1       #show

  ​ print('grabbing the first element')

  ​ pool1['USA']  # first label index

Out[]: ​ USA        1

  ​ Britain    2

  ​ France     3

  ​ Germany    4

  ​ dtype: int64

  ​ grabbing the first element

Out[]: 1

As shown in the code above, to grab a series element, use the same approach as the numpy array indexing, but by passing the label corresponding to that data point. The data type of the label is also important, notice the ‘USA’ label was passed as a string to grab the data point ‘1’. If the label is numeric, then the indexing would be similar to that of a numpy array. Consider numeric indexing in the following example:

In []: pool2 = pd.Series(['USA','Britain','France','Germany'],[1,2,3,4])

  ​ pool2      #show

  ​ print('grabbing the first element')

  ​ pool2[1]  #numeric indexing

Out[]: 1        USA

  ​ 2    Britain

  ​ 3     France

  ​ 4    Germany

  ​ dtype: object

  ​ grabbing the first element

Out[]: 'USA'

Tip: you can easily know the data held by a series through the dtype. Notice how the dtype for pool1 and pool2 are different, even though they were both created from the same lists. The difference is that pool2 holds strings as its data points, while pool1 holds integers (int64).

Panda series can be added together. It works best if the two series have similar labels and data points.

Example 70 : Adding series

Let us create a third series ‘pool 3’. This is a similar series as pool1, but Britain has been replaced with ‘USSR’, and a corresponding data point value of 5.

In []: pool3 = pd.Series([1,5,3,4],['USA','USSR','France',
'Germany'])

  ​ pool3

Out[]: USA        1

  ​ USSR       5

  ​ France     3

  ​ Germany    4

  ​ dtype: int64

Now adding series:

In []: ​ # Demonstrating series addition

  ​ double_pool = pool1 + pool1

  ​ print('Double Pool')

  ​ double_pool

  ​ mixed_pool = pool1 + pool3

  ​ print('\nMixed Pool')

  ​ mixed_pool

  ​ funny_pool = pool1 + pool2

  ​ print('\nFunny Pool')

  ​ funny_pool

  ​ Double Pool

Out[]: USA        2

  ​ Britain    4

  ​ France     6

  ​ Germany    8

  ​ dtype: int64

  ​ Mixed Pool

Out[]: Britain    NaN

  ​ France     6.0

  ​ Germany    8.0

  ​ USA        2.0

  ​ USSR       NaN

  ​ dtype: float64

  ​ Funny Pool

C:\Users\Oguntuase\Anaconda3\lib\site-     packages\pandas\core\indexes\base.py:3772: RuntimeWarning: '<' not supported between instances of 'str' and 'int', sort order is undefined for incomparable objects

  ​ return this.join(other, how=how, return_indexers=return_indexers)

Out[]: USA        NaN

  ​ Britain    NaN

  ​ France     NaN

  ​ Germany    NaN

  ​ 1          NaN

  ​ 2          NaN

  ​ 3          NaN

  ​ 4          NaN

  ​ dtype: object

By adding series, the resultant is the increment in data point values of similar labels (or indexes). A ‘NaN’ is returned in instances where the labels do not match.

Notice the difference between Mixed_pool and Funny_pool: In a mixed pool, a few labels are matched, and their values are added together (due to the add operation). For Funny_pool, no labels match, and the data points are of different types. An error message is returned and the output is a vertical concatenation of the two series with ‘NaN’ Datapoints.

Tip: As long as two series contain the same labels and data points of the same type, basic array operations like addition, subtraction, etc. can be done. The order of the labels does not matter, the values will be changed based on the operator being used. To fully grasp this, try running variations of the examples given above.

Data frames

A Pandas data frame is just an ordered collection of Pandas series with a common/shared index. At its basic form, a data frame looks more like an excel sheet with rows, columns, labels and headers. To create a data frame, the following syntax is used:

pd.DataFrame(data=None, index=None, columns=None, dtype=None, copy=False)

Usually, the data input is an array of values (of whatever datatype). The index and column parameters are usually lists/vectors of either numeric or string type.

If a Pandas series is passed to a data frame object, the index automatically becomes the columns, and the data points are assigned accordingly.

Example 71 : Creating a data frame

In []: df = pd.DataFrame([pool1])        # passing a series

 df                                # show

 # two series

 index = 'WWI WWII'.split()

 new_df = pd.DataFrame([pool1,pool3],index)

 new_df                            # show

Output:

USA

Britain

France

Germany

0

1

2

3

4

USA

Britain

France

Germany

WWI

1

2

3

4

WWII

5

1

3

4

We have created two data frames from the pool 1 and pool 3 series we created earlier. Notice how the first data frame assigns the series labels as column headers, and since no index was assigned, a value of ‘0’ was set at that index i.e. row header.

For the second data frame, the row labels were specified by passing a list of strings [‘WWI’,’WWII’].

Tip: The .split( ) string method is a quick way of creating lists of strings. It works by splitting a string into its component characters, depending on the delimiter passed to the string method.

For example, let us split this email ‘pythonguy@gmail.com’ into a list containing the username and the domain name.

In []:  # Illustrating the split() method

  ​ email = 'pythonguy@gmail.com'

  ​ string_vec = email.split('@')

   ​ string_vec     # show

  ​ A = string_vec[0]; B = string_vec[1] # Extracting values

  ​ print('Username:',A,'\nDomain name:',B)

Out[]:  ['pythonguy', 'gmail.com']

      ​ Username: pythonguy

      ​ Domain name: gmail.com

To create a data frame with an array, we can use the following method:

# Creating dataframe with an array

Array = np.arange(1,21).reshape(5,4)  # numpy array

row_labels = 'A B C D E'.split()

col_labels = 'odd1 even1 odd2 even2'.split()

Arr_df = pd.DataFrame(Array,row_labels,col_labels)

Arr_df

Output:

odd1

even1

odd2

even2

A

1

2

3

4

B

5

6

7

8

C

9

10

11

12

D

13

14

15

16

E

17

18

19

20

Notice how this is not unlike how we create spreadsheets in excel. Try playing around with creating data frames.

Exercise: Create a data frame from a 5 4 array of uniformly distributed random values. Include your choice row and column names using the .split( ) method.

Hint: use the rand function to generate your values, and use the reshape method to form an array.

Now that we can conveniently create Data frames, we are going to learn how to index and grab elements off them.

Tip: Things to note about data frames.

They are a collection of series (more like a list with Pandas series as its elements).

They are similar to numpy arrays i.e. they are more like n m dimensional matrices, where ‘n’ are the rows and ‘m’ are the columns.

Example 72 : Grabbing elements from a data frame.

The easiest elements to grab are the columns. This is because, by default, each column element is a series with the row headers as labels. We can grab them by using a similar method from the series – indexing by name.

In []: # Grab data frame elements

  ​ Arr_df['odd1']    # grabbing first column

Out[]: A     1

  ​ B     5

  ​ C     9

  ​ D    13

  ​ E    17

  ​ Name: odd1, dtype: int32

Pretty easy, right? Notice how the output is like a Pandas series. You can verify this by using the type(Arr_df[ 'odd1' ] ) command.

When more than one column is grabbed, however, it returns a data frame (which makes sense, since a data frame is a collection of at least two series). To grab more than one column, pass the column names to the indexing as a list. This is shown in the example code below:

In []: ​ # Grab two columns

  ​ Arr_df[['odd1','even2']] # grabbing first and last columns

Output:

odd1

even2

A

1

4

B

5

8

C

9

12

D

13

16

E

17

20

To select a specific element, use the double square brackets indexing notation we learned under array indexing. For example, let us select the value 15 from Arr_df.

In []: Arr_df['odd2']['D']

Out[]: 15

You may decide to break the steps into two, if it makes it easier. This method is however preferred as it saves memory from variable allocation. To explain, let us break it down into two steps.

In []: x = Arr_df['odd2']

  ​ x

Out[]: A     3

  ​ B     7

  ​ C    11

  ​ D    15

  ​ E    19

  ​ Name: odd2, dtype: int32

See that the first operation returns a series containing the element ‘15’. This series can now be indexed to grab 15 using the label ‘D’.

In []: x['D']

Out[]: 15

While this approach works, and is preferred by beginners, a better approach is to get comfortable with the first method to save coding time and resources.

To grab rows, a different indexing method is used.
You can use either data_frame_name.loc[‘row_name’ ] or data_frame_name.iloc[‘row_index’ ] .

Let us grab the row E from Arr_d f .

In []: print("using .loc['E']")

      Arr_df.loc['E']

      print('\nusing .iloc[4]')

      Arr_df.iloc[4]

        using .loc['E']

Out[]:

      odd1     17

      even1    18

      odd2     19

      even2    20

      Name: E, dtype: int32

        using .iloc[4]

Out[]:

      odd1     17

      even1    18

      odd2     19

      even2    20

      Name: E, dtype: int32

See, the same result!

You can also use the row indexing method to select single items.

In []: Arr_df.loc['E']['even2']

  ​ # or

  ​   Arr_df.iloc[4]['even2']

Out[]: 20

Out[]: 20

Moving on, we will try to create new columns in a data frame, and also delete a column.

In []: # Let us add two sum columns to Arr_df

  ​ Arr_df['Odd sum'] = Arr_df['odd1']+Arr_df['odd2']

  ​ Arr_df['Even sum'] = Arr_df['even1']+Arr_df['even2']

 Arr_df

Output:

odd1

even1

odd2

even2

Odd sum

Even sum

A

1

2

3

4

4

6

B

5

6

7

8

12

14

C

9

10

11

12

20

22

D

13

14

15

16

28

30

E

17

18

19

20

36

38

Notice how the new columns are declared. Also, arithmetic operations are possible with each element in the data frame, just like we did with the series.

Exercise: Add an extra column to this data frame. Call it Total Sum, and it should be the addition of Odd sum and Even sum.

To remove a column from a data frame, we use the data_frame_name.drop( ) method.

Let us remove the insert a new column and then remove it using the .drop( ) method.

In []: Arr_df['disposable'] = np.zeros(5)   # new column

      Arr_df   #show

Output:

odd1

even1

odd2

even2

Odd sum

Even sum

disposable

A

1

2

3

4

4

6

0.0

B

5

6

7

8

12

14

0.0

C

9

10

11

12

20

22

0.0

D

13

14

15

16

28

30

0.0

E

17

18

19

20

36

38

0.0

To remove the unwanted column:

In []: # to remove

  ​ Arr_df.drop('disposable',axis=1,inplace=True)

  ​ Arr_df

Output:

odd1

even1

odd2

even2

Odd sum

Even sum

A

1

2

3

4

4

6

B

5

6

7

8

12

14

C

9

10

11

12

20

22

D

13

14

15

16

28

30

E

17

18

19

20

36

38

Notice the ‘axis=1’ and ‘inplace = True’ arguments. These are parameters that specify the location to perform the drop i.e. axis (axis = 0 specifies row operation), and intention to broadcast the drop to the original data frame, respectively. If ‘inplace= False’, the data frame will still contain the dropped column.

Tip: The ‘inplace = False’ method is used for assigning an array to another variable without including certain columns.

Conditional selection

Similar to how we conditional selection works with NumPy arrays, we can select elements from a data frame that satisfy a Boolean criterion.

You are expected to be familiar with this method, hence, it will be done in one step.

Example 72: Let us grab sections of the data frame ‘Arr_df’ where the value is > 5.

In []: # Grab elements greater than five

Arr_df[Arr_df>5]

Output:

odd1

even1

odd2

even2

Odd sum

Even sum

A

NaN

NaN

NaN

NaN

NaN

6

B

NaN

6.0

7.0

8.0

12.0

14

C

9.0

10.0

11.0

12.0

20.0

22

D

13.0

14.0

15.0

16.0

28.0

30

E

17.0

18.0

19.0

20.0

36.0

38

Notice how the instances of values less than 5 are represented with a ‘NaN’.

Another way to use this conditional formatting is to format based on column specifications.

You could remove entire rows of data, by specifying a Boolean condition based off a single column. Assuming we want to return the Arr_df data frame without the row ‘C’.  We can specify a condition to return values where the elements of column ‘odd1’ are not equal to ‘9’ (since row C contains 9 under column ‘odd1’).

In []: # removing row C through the first column

      Arr_df[Arr_df['odd1']!= 9]

Output:

odd1

even1

odd2

even2

Odd sum

Even sum

A

1

2

3

4

4

6

B

5

6

7

8

12

14

D

13

14

15

16

28

30

E

17

18

19

20

36

38

Notice that row ‘C’ has been filtered out. This can be achieved through a smart conditional statement through any of the columns.

In []:  # does the same thing : remove row ‘C’

       # Arr_df[Arr_df['even2']!= 12]

In[]: # Let us remove rows D and E through 'even2'

       Arr_df[Arr_df['even2']<= 12]

Output

odd1

even1

odd2

even2

Odd sum

Even sum

A

1

2

3

4

4

6

B

5

6

7

8

12

14

C

9

10

11

12

20

22

Exercise: Remove rows C, D, E via the ‘Even sum’ column. Also, try out other such operations as you may prefer.

To combine conditional selection statements, we can use the ‘logical and, i.e. &’, and the ‘logical or, i.e. |’ for nesting multiple conditions. The regular ‘and’ and ‘or’ operators would not work in this case as they are used for comparing single elements. Here, we will be comparing a series of elements that evaluates to true or false, and those generic operators find such operations ambiguous.

Example 73: Let us select elements that meet the criteria of being greater than 1 in the first column, and less than 22 in the last column. Remember, the ‘and statement’ only evaluates to true if both conditions are true.

In []:​ Arr_df[(Arr_df['odd1']>1) & (Arr_df['Even sum']<22)]

Output:

odd1

even1

odd2

even2

Odd sum

Even sum

B

5

6

7

8

12

14

Only the elements in Row ‘B’ meet this criterion, and were returned in the data frame.

This approach can be expounded upon to create even more powerful data frame filters.

Missing Data

There are instances when the data being imported or generated into pandas is incomplete or have missing data points. In such a case, the likely solution is to remove such values from the dataset, or to fill in new values based on some statistical extrapolation techniques. While we would not be fully exploring statistical measures of extrapolation (you can read up on that from any good statistics textbook), we would be considering the use of the .dropna( ) and .fillna( ) methods for removing and filling up missing data points respectively.

To illustrate this, we will create a data frame – to represent imported data with missing values, and then use these two data preparation methods on it.

Example 73: Another way to create a data frame is by using a dictionary. Remember, a python dictionary is somehow similar to a Pandas series in that they have key-value pairs, just as Pandas series are label-value pairs (although this is a simplistic comparison for the sake of conceptualization).

In []: ​ # First, our dictionary

  ​ dico = {'X':[1,2,np.nan],'Y':[4,np.nan,np.nan],'Z':[7,8,9]}

  ​ dico #show

  ​ # passing the dictionary to a dataframe

  ​ row_labels = 'A B C'.split()

  ​ df = pd.DataFrame(dico,row_labels)

  ​ df #show

Output:

{'X': [1, 2, nan], 'Y': [4, nan, nan], 'Z': [7, 8, 9]}

X

Y

Z

A

1.0

4.0

7

B

2.0

NaN

8

C

NaN

NaN

9

Now, let us start off with the .dropna( ) method. This removes any ‘null’ or ‘nan’ values in the data frame it’s called off, either column-wise or row-wise, depending on the axis specification and other arguments passed to the method. It has the following default syntax:

df.dropna(axis=0, how='any', thresh=None, subset=None, inplace=False)

The ‘df’ above is the data frame name. The default axis is set to zero, which represent row-wise operation. Hence, at default, the method will remove any row containing ‘nan’ values.

Let us see what happens when we call this method for our data frame.

In []:  # this removes ‘nan’ row-wise

       df.dropna()

Output:

X

Y

Z

A

1.0

4.0

7

Notice that rows B and C contain at least a ‘nan’ value. Hence, they were removed.

Let us try a column-wise operation by specifying the axis=1.

In []:  # this removes ‘nan’ column-wise

       df.dropna(axis=1)

Output:

Z

A

7

B

8

C

9

As expected, only the column ‘Z’ was returned.

Now, in case we want to set a condition for a minimum number of ‘non-nan’ values/ actual data points required to make the cut (or escape the cut, depending on your perspective), we can use the ‘thresh’ (short for threshold) parameter to specify this.

Say, we want to remove ‘nan’ row-wise, but we only want to remove instances where the row had more than one actual data point value. We can set the threshold to 2 as illustrated in the following code:

In []: # drop rows with less than 2 actual values

       df.dropna(thresh = 2)

Output:

X

Y

Z

A

1.0

4.0

7

B

2.0

NaN

8

Notice how we have filtered out row C, since it contains only one actual value ‘9’.

Exercise: Filter out columns in the data frame ‘df’ containing less than 2 actual data points

Next, let us use the .fillna( ) method to replace the missing values with our extrapolations.

This method has the following syntax:

df.fillna(value=None, method=None, axis=None, inplace=False, limit=None, downcast=None, **kwargs)

Tip: Reminder, you can always use shift + ta b to check the documentation of methods and functions to know their syntax.

Let us go ahead and replace our ‘NaN’ values with an ‘x’ marker. We can specify the ‘X’ as a string, and pass it into the ‘value’ parameter in .fillna( ) .

In []: # filling up NaNs

 df.fillna('X')

Output:

X

Y

Z

A

1

4

7

B

2

X

8

C

X

X

9

While marking missing data with an ‘X’ is fun, it is sometimes more intuitive (for lack of a better statistical approach), to use the mean of the affected column as a replacement for the missing elements.

Example 74 : Filling up missing data.

Let us first use the mean method to fill up column ‘X’, then based off that simple step, we will use a for loop to automatically fill up missing data in the data frame.

In []: # Replacing missing values with mean in column ‘X’

  ​ df['X'].fillna(value = df['X'].mean())

Out[]: A    1.0

  ​ B    2.0

  ​ C    1.5

  ​ Name: X, dtype: float64

Notice that the value of the third element in column ‘X’ has changed to 1.5. This is the mean of that column. The one line code that accomplished this could have been broken down into multiple line for better understanding. This is shown below:

In []: ​ # variables

  ​ xcol_var = df['X']

  ​ xcol_mean = xcol_var.mean()  # or use mean(xcol_var)

  ​ # instruction

  ​ xcol_var.fillna(value = xcol_mean)

Out[]: A    1.0

  ​ B    2.0

  ​ C    1.5

  ​ Name: X, dtype: float64

Same results, but more coding and more memory use via variable allocation.

Now, let us automate the entire process.

In []: for i in 'X Y Z'.split():              # loop

      df[i].fillna(value = df[i].mean(),inplace=True)

      df                # show

X

Y

Z

A

1.0

4.0

7

B

2.0

NaN

8

C

NaN

NaN

9

Output:

X

Y

Z

A

1.0

4.0

7

B

2.0

4.0

8

C

1.5

4.0

9

New data frame                                                Old data frame

While the output only displays the data frame on the left, the original data frame is put here for comparison. Notice the new values replacing the NaNs. For the column ‘Y’, the mean is 4.0, since that is the only value present.

This is a small operation that can be scaled for preparing and formatting larger datasets in Pandas.

Tip: The other arguments of the .fillna( ) method can be explored, including the fill methods: for example, forward-fill - which fills the missing value with the previous row/column value based on the value of the limit parameter i.e. if limit = 1, it fills the next 1 row/column with the previous row/column value; also, the back-fill - which does the same as forward-fill, but backwards.

Group-By

This Pandas method, as the name suggests, allows the grouping of related data to perform combined/aggregate operations on them.

Example 75: Creating a data frame of XYZ store sales.

In []: # Company XYZ sales information

       # Dictionary containing needed data

  ​ data = {'Sales Person':'Sam Charlie Amy Vanessa Carl Sarah'.split(),

         ​   'Product':'Hp Hp Apple Apple Dell Dell'.split(),

         ​   'Sales':[200,120,340,124,243,350]}

  ​ print('XYZ sales information\n________________')  # print info.

  ​ serial = list(range(1,7))         # row names from 1-6

  ​ df = pd.DataFrame(data,serial)   # build data frame

  ​ df

Output:  

XYZ sales information

_________________________________________________

Sales Person

Product

Sales

1

Sam

Hp

200

2

Charlie

Hp

120

3

Amy

Apple

340

4

Vanessa

Apple

124

5

Carl

Dell

243

6

Sarah

Dell

350

From our dataset, we can observe some common items under the product column. This is an example of an entry point for the group-by method in a data set.  We can find information about the sales using the product grouping.

In []: # finding sales information by product

  ​ print('Total items sold: by product')

  ​ df.groupby('Product').sum()

Total items sold: by product

Sales

Product

Apple

464

Dell

593

Hp

320

This is an example of an aggregate operation using groupby. Other functions can be called to display interesting results as well. For example, .count( ) :

In []:  df.groupby('Product').count()

Output:

Sales Person

Sales

Product

Apple

2

2

Dell

2

2

Hp

2

2

While the previous operation could not return the ‘Sales person’ field, since a numeric operation like ‘sum’ cannot be performed on a string, the count method returns the instances of each product within both categories. Via this output, we can easily infer that XYZ company assigns two salespersons per product, and that each of the sales persons made a sale of the products. However, unlike the sum method, this count method does not give a clearer overview of the sales. This is why so many methods are usually called to explain certain aspects of data. A very useful method for checking multiple information at a time is the .describe() method.

In []:  #Getting better info using describe ()

  ​    df.groupby('Product').describe()

Output:

Sales

count

mean

std

min

25%

50%

75%

max

Product

Apple

2.0

232.0

152.735065

124.0

178.00

232.0

286.00

340.0

Dell

2.0

296.5

75.660426

243.0

269.75

296.5

323.25

350.0

Hp

2.0

160.0

56.568542

120.0

140.00

160.0

180.00

200.0

Now, this is more informative. It says a lot about the data at a glance. Individual products can also be selected: df.groupby(‘Product’).describe()[‘Product name e.g. ‘Apple’ ] .

Concatenate, Join and Merge

These are methods for combining multiple data frames, or data sets into a single one. They differ in syntax, and achieve specific combinations of data frames based on the intended output.

Concatenation allows datasets to be ‘glued’ together, either row-wise or column-wise. Here, the dimensions of the data frames must be the same along the axis of concatenation, i.e. row-wise concatenation requires the two data frames to have the same number of columns, and vice versa.

Example 76:    Let us create two example data frames and use the concatenate method.

In []: ​ # Defining a dictionary of values

  ​ d1 = {'A':'A1 A2 A3'.split(),'B':'B1 B2 B3'.split(),

      ​ ​ 'C':'C1 C2 C3'.split()}

  ​ d2 = {'A':'A4 A5 A6'.split(),'B':'B4 B5 B6'.split(),

      ​ ​ 'C':'C4 C5 C6'.split()}

  ​ # Now the data frames

  ​ df1 = pd.DataFrame(d1,index= '1 2 3'.split())

  ​ df2 = pd.DataFrame(d2,index= '4 5 6'.split())

  ​ # concatenating

  ​ pd.concat([df1,df2])    # row-wise axis = 0

  ​ pd.concat([df1,df2],axis=1,sort=True) # col-wise axis = 1

Out[]:

A

B

C

1

A1

B1

C1

2

A2

B2

C2

3

A3

B3

C3

4

A4

B4

C4

5

A5

B5

C5

6

A6

B6

C6

Out[]:

A

B

C

A

B

C

1

A1

B1

C1

NaN

NaN

NaN

2

A2

B2

C2

NaN

NaN

NaN

3

A3

B3

C3

NaN

NaN

NaN

4

NaN

NaN

NaN

A4

B4

C4

5

NaN

NaN

NaN

A5

B5

C5

6

NaN

NaN

NaN

A6

B6

C6

Notice how the result of the row-wise concatenation is just like placing the first data frame on top of the other. The attempt at column concatenation for these data frames resulted in NaNs because they have varying indices. While df1 has its index from 1-3, with corresponding values; df2 has its own index ranging from 4-6. To allow for proper column concatenation, we have to specify a similar index as below.

In []: df2.index = '1 2 3'.split() # setting same index as df1

  ​ # This should then work

  ​ pd.concat([df1,df2],axis=1) # col-wise axis = 1

Out[21]:

A

B

C

A

B

C

1

A1

B1

C1

A4

B4

C4

2

A2

B2

C2

A5

B5

C5

3

A3

B3

C3

A6

B6

C6

Now, this is a nicely combined data set. This is more likely the kind of concatenation you would encounter while working with data sheets, as it allows for a more flexible naming for the columns, while sharing similar row-indices.

Exercise: Change the last three column names in the above output table to ‘D E F’, i.e. the table should now have columns ‘A B C D E F’.

Hint: use df2.columns = ‘value ’ to change the column names.

Merge and Join on the other hand, are more similar. They take two data frames and combine them together column-wise. They both require a specification for a right and left data frame respectively to determine the organization. Also, they require an extra specification for an entry point (usually called a key) where the join or merge operation occurs. While the Merge and Join methods are powerful, we will not be exploring them.

Reading and Writing data

In real-world applications, data comes in various formats. These are the most common ones: CSV, Excel spreadsheets (xlsx / xls), HTML and SQL. While Pandas can read SQL files, it is not necessarily the best for working with SQL databases, since there are quite a few SQL engines: SQL lite, PostgreSQL, MySQL, etc. Hence, we will only be considering CSV, Excel and HTML.

Read

The pd.read_file_type(‘file_name’ ) method is the default way to read files into the Pandas framework. After import, pandas displays the content as a data frame for manipulation using all the methods we have practiced so far, and more.

CSV (comma separated variables) & Excel

Create a CSV file in excel and save it in your python directory. You can check where your python directory is in Jupyter notebook by typing: pwd() . If you want to change to another directory containing your files (e.g. Desktop), you can use the following code:

In []: import os

  ​ os.chdir('C:\\Users\\Username\\Desktop')

To import your CSV file, type: pd.read_csv(‘csv_file_name’). Pandas will automatically detect the data stored in the file and display it as a data frame. A better approach would be to assign the imported data to a variable like this:

In []: ​ Csv_data = pd.read_csv(‘example file.csv’)

      ​ Csv_data             # show

Running this cell will assign the data in ‘example file.csv’ to the variable Csv_data, which is of the type data frame. Now it can be called later or used for performing some of the data frame operations.

For excel files (.xlsx and .xls files), the same approach is taken. To read an excel file named ‘class data.xlsx’, we use the following code:

In []: ​ Xl_data = pd.read_excel(‘class data.xlsx’)

     ​ Xl_data             # show

This returns a data frame of the required values. You may notice that an index starting from 0 is automatically assigned at the left side. This is similar to declaring a data frame without explicitly including the index field. You can add index names, like we did in previous examples.

Tip: in case the excel spreadsheet has multiple sheets filled. You can specify the sheet you need to be imported. Say we need only sheet 1, we use:  sheetname = ‘Sheet 1 ’ . For extra functionality, you may check the documentation for read_excel( ) by using shift+ta b .

Write

After working with our imported or pandas-built data frames, we can write the resulting data frame back into various formats. We will, however, only consider writing back to CSV and excel. To write a data frame to CSV, use the following syntax:

In []:​ Csv_data.to_csv(‘file name ’,index = False)

This writes the data frame ‘Csv_data’ to a CSV file with the specified filename in the python directory. If the file does not exist, it creates it.

Tip: You can also use this method to create spreadsheet files via Python. The False index parameter is to ensure that the automatic indexing of Jupyter is not written to the file, thereby messing up its formatting.

For writing to an excel file, a similar syntax is used, but with sheet name specified for the data frame being exported.

In []:   Xl_data.to_excel(‘file name.xlsx’,sheet_name = ‘Sheet 1’)

This writes the data frame Xl_dat a to sheet one of ‘file name.xlsx ’ .

Html

Reading Html files through pandas requires a few libraries to be installed: htmllib5, lxml, and BeautifulSoup4. Since we installed the latest Anaconda, these libraries are likely to be included. Use conda lis t to verify, and conda instal l to install any missing ones.

Html tables can be directly read into pandas using the pd.read_html (‘sheet url’ ) method. The sheet url is a web link to the data set to be imported. As an example, let us import the ‘Failed bank lists’ dataset from FDIC’s website and call it w_data.

In []: w_data = pd.read_html('http://www.fdic.gov/bank/individual/failed/banklist.html')

 w_data[0]

To display the result, here we used w_data [0 ] . This is because the table we need is the first sheet element in the webpage source code. If you are familiar with HTML, you can easily identify where each element lies. To inspect a web page source code, use Chrome browser. On the web page >> right click >> then, select ‘view page source ’ . Since what we are looking for is a table-like data, it will be specified like that in the source code. For example, here is where the data set is created in the FDIC page source code:

FDIC page source via chrome

This section concludes our lessons on the Pandas framework. To test your knowledge on all that has been introduced, ensure to attempt all the exercises below. In the next chapter, we will be exploring some data visualization frameworks.

For the exercise, we will be working on an example dataset. A salary spreadsheet from Kaggle.com. Go ahead and download the spreadsheet from this link: www.kaggle.com/kaggle/sf-salaries

Note: You might be required to register before downloading the file. Download the file to your python directory and extract the file.

Exercises: We will be applying all we have learned here.

Import pandas as pd

Import the CSV file into Jupyter notebook, assign it to a variable ‘Sal’, and display the first 5 values.

Hint: use the .head( ) method to display the first 5 values of a data frame. Likewise, .tail( ) is used for displaying the last 5 results. To specify more values, pass ‘n=value’ into the method.

What is the highest pay (including benefits)?    Answer: 567595.43

Hint: Use data frame column indexing and .max( ) method.

According to the data, what is ‘MONICA FIELDS’s Job title, and how much does she make plus benefits? Answer:  Deputy Chief of the Fire Department, and $ 261,366.14 .

Hint: Data frame column selection and conditional selection works (conditional selection can be found under Example 72. Use column index ==’string’ for the Boolean condition).

Finally, who earns the highest basic salary (minus benefits), and by how much is their salary higher than the average basic salary. Answer: NATHANIEL FORD earns the highest. His salary is higher than the average by $ 492827.1080282971.

Hint: Use the .max( ) , and .mean( ) methods for the pay gap. Conditional selection with column indexing also works for the employee name with the highest pay.

Best of luck.

Chapter 3

Data Visualization with Python

Data visualization can be described as the various ways by which analyzed data i.e. information is displayed. Sometimes, even well-analyzed data is not informative enough at a glance. With data visualization, which includes line graphs, bar charts, pictograms, etc. the results/ analysis being presented become less abstract to the end-user, and decision-making is enhanced. In this chapter, we will be learning various techniques for displaying the results of our analysis with NumPy and Pandas frameworks.

Matplotlib

This is a python library for producing high-quality 2D plots. For those that have some MATLAB experience, the plotting techniques and visualizations here will seem familiar. Matplotlib offers a lot of flexibility with plots, in terms of control over things like the axes, fonts, line styles and size, etc. However, all these require writing extra lines of code. So, if you do not mind going the extra mile (with typing code) to fully specify your plots, then matplotlib is your friend. For extra information about this package, visit the official page at www.matplotlib.org

There are basically two approaches to plotting data in matplotlib: The Functional and the object-oriented (OO) approach, respectively. You might encounter the two terms consistently as you interact with programmers and other programming languages, but they are just two slightly different approaches to programming. We will only be considering the functional approach here, as it is easy to understand for beginners and also requires writing fewer lines of code. The OO method, however, offers more control over the plots at the consequence of writing more lines of code. 

To start off, let us create a simple cosine plot using the functional approach.

First, let us import the relevant libraries and create plot data:

In []: import matplotlib.pyplot as plt 

      import numpy as np

      %matplotlib inline

      # creating plot values

      x = np.linspace(0,10)  # x-axis/time-scale

      y = np.cos(x)    # corresponding cosine values

The %matplotlib inlin e option in the code ensures all our plots are displayed as we run each cell. If you are running a different python console, you can put plt.show( ) at the end of your code to display your plots. plt.show( ) is the print( ) function equivalent for matplotlib plots.

Functional method

In []: # functional plot

  ​ plt.plot(x,y)

Out[]: [<matplotlib.lines.Line2D at 0x25108cf27f0>]

Notice that we get an Out[ ] statement. This is because we did not print the result using plt.show( ) . While this is not significant if you are using Jupyter, it might be required for other consoles.

We can also plot multiple functions in one graph.

In []: ​ z = np.sin(x)  # Adding an extra plot variable

  ​ plt.plot(x,y,x,z);plt.show()

To print multiple graphs, just pass each argument to the plot statement, and separate the plots with commas.

To make our graphs more meaningful, we can label the axes and give the graph a title.

In []: plt.plot(x,y,x,z)

 plt.xlabel('Time axis')  # labelling x-axis

 plt.ylabel('Magnitude')  # labelling y-axis

 plt.title('Sine and Cosine waves')  # graph title

 plt.show()  # printing

Output:

Now, this is a better figure. Since we added more than one plot, there is an extra functionality called ‘legend’ which helps differentiate between the plots.

The legend function takes in two arguments. The first is usually a string argument for labeling the graphs in order. The second is for extra functionality, like where the legend should be. The location argument is specified using ‘loc=value’. For value = 1, upper right corner; 2, for upper left corner; 3, for lower left corner; and 4 for the lower right corner. However, if you would rather let matplotlib decide the best location, use ‘value=0’.

In []: plt.plot(x,y,x,z)

  ​ plt.xlabel('Time axis')

  ​ plt.ylabel('Magnitude')

  ​ plt.title('Sine and Cosine waves')

  ​ plt.legend(['y','z'],loc=0)    # loc=0 means best location

  ​ plt.show()

Output:

Assuming we wish to plot both the cosine and sine wave above, but side by side.

We can use the subplot command to do this. Think of the subplot as an array of figures with a specification of the number of rows and columns. So, if we want just two graphs beside each otherhat can be considered as a 1 row, and 2 columns array.

In []: ​ # subplotting

  ​ plt.subplot(1,2,1)    # plot 1

  ​ plt.plot(x,y)

  ​ plt.title('Cosine plot')

  ​ plt.subplot(1,2,2)     # plot 2

  ​ plt.plot(x,z)

  ​ plt.title('Sine plot')

  ​ plt.tight_layout()   # avoid overallping plots

  ​ plt.show()

Tip: The tight_layout() line ensures that all subplots are well spaced. Always use this when sub-plotting to make your plots nicer. Try removing that line and compare the output!

Output:

To explain the subplot line i.e. subplot(1,2,1) and subplot(1,2,2): The first two values are the number of rows and columns of the subplot. As seen in the above result, the plot is on one row, and two columns. The last value specifies the order of the plot; hence (1,2,1) translates to plotting that figure in the first row, and the first column out of two.

We can specify the line colors in our plots, as well as the line style. This is passed as a string after the plot argument. Since all the plot options, including marker style and the likes are exactly same as for Matlab, here is a link to Matlab plot documentation to explore all the extra customization feature you can port to your matplotlib plots: https://www.mathworks.com/help/matlab/ref/plot.html#btzpndl-1

Let us change the color and fonts in our subplots to illustrate this.

In []: plt.subplot(1,2,1)    # plot 1

 plt.plot(x,y,'r-x')   # red plot with -x marker

 plt.title('Cosine plot')

 ​ plt.subplot(1,2,2)    # plot 2

 plt.plot(x,z,'g-o')   # green plot with -o marker

 plt.title('Sine plot')

 plt.tight_layout()   # still avoiding overallping plots

 plt.show()

Output:

Exercise: Now that you have learned how to plot using the functional approach, test your skills.

Plot this:

Tip: use np.arange(0,10,11 ) for the x-axis.

After creating your plot, you may need to import it into your documents or just save it on your device. To do this, you can right-click the image in your Jupyter notebook and click copy. Then, you can paste the copied image into your document.

If you prefer to save, you can use the plt.savefig(‘figurename.extension’, DPI = value ) method. Here, figurename is the desired name for your saved image; The extension is the desired format i.e. PNG, JPG, BMP, etc. Finally, the DPI specifies the quality of the image, the higher the better. Usually for standard printing quality, about 300 is good enough.

Tip: Learning a bit more about these specifications can really help you generate better images from your plots.

Seaborn

This is another data visualization library that extends the graphical range of the matplotlib library. A lot of methods from matplotlib are applicable here, for customizing plots. However, it generates high quality, dynamic plots in fewer lines of code.

Seaborn is more optimized for plotting trends in datasets, and we are going to explore a dataset using this library. Since, Seaborn is pre-loaded with a few datasets (it can call and load certain datasets from its online repository), we will just load up one of these for our example.

Example 77 : Loading a seaborn dataset and plotting trends

As with other packages, we have to import seaborn using the standard variable name ‘sns’.

In []: # importing seaborn

  ​ import seaborn as sns

  ​ %matplotlib inline

Next, we will load the popular ‘tips’ dataset from Seaborn. This dataset contains information about a restaurant, the tips given to the waiters, amount of tip, size of the customer group (e.g. a party of 3 people) etc.

In []: ​ # loading a dataset from seaborn

  ​ tips_dataset = sns.load_dataset('tips')

  ​ tips_dataset.head()

Out[]:

total_bill

tip

sex

smoker

day

time

size

0

16.99

1.01

Female

No

Sun

Dinner

2

1

10.34

1.66

Male

No

Sun

Dinner

3

2

21.01

3.50

Male

No

Sun

Dinner

3

3

23.68

3.31

Male

No

Sun

Dinner

2

4

24.59

3.61

Female

No

Sun

Dinner

4

Next, we can find trends in the data using different kinds of plots. Let us use the dist_plot (distribution plot) to observe how the total_bill is distributed across the dataset.

In []: ​ #Use distplot

  ​ sns.distplot(tips_dataset['total_bill'],bins = 30,kde=False)

Out[]:<matplotlib.axes._subplots.AxesSubplot at 0x28d3867d6d8>

The distplot shows the distribution of certain trends within data. According to the plot above, we can infer that most of the bills are within the range of $10 - $20, since they have the tallest bins within the distribution. You may notice some extra arguments in the distplot code, the bins argument controls the number of histograms that are shown within the population. The higher the value, the more histograms. Although, sometimes higher can make the data less obvious to read, so finding a balance is important. The next argument is the ‘kde’, which means kernel density estimate. It is sometimes preferred over histograms, or along with histograms for a more accurate interpretation of the data. It is mostly an estimate of the probability density function of any variable within the distribution, more like the histogram but smoother. You can read up for a more thorough statistical background on some of these things.

Another useful plot is the relplot, which shows the relationship between two variables within a data_set. It is good for comparison, and can sort results based off categories i.e. sex, age, etc. within the data.

Let us demonstrate how the total_bill relates with the estimated tip, and sort by category male/female.

In []: # estimating tip with respect to total_bill

  ​ sns.relplot(x ="total_bill", y="tip", data = tips_dataset)

out []:

Here is a basic plot without the category argument. This tells us that the tips generally increase with respect to the total bill. The higher bills correspond to higher tips, and lower bills to lower tips. Adding categories makes the data more interesting as we can see the category that tips more or less.

In []: sns.relplot(x ="total_bill", y="tip", data = tips_dataset, hue = 'sex')

Out[]: <seaborn.axisgrid.FacetGrid at 0x28d3b9cedd8>

See how this is a more informative data. This tells how male customers tipped higher per total_bill on average than the females.

Now this idea can be extended and applied to more advanced data. You can further explore various plotting options with relplot via this link: www.seaborn.pydata.org/tutorial/relational.html

The relplot is even extended with the pairplot option, which relates everything in a data set in one plot. It is a great way to get a quick overview of important trends within your data.

In []: sns.pairplot(tips_dataset,hue = 'sex', palette = 'coolwarm')

Out[]:<seaborn.axisgrid.PairGrid at 0x28d412f2e48>

See how easy it is to observe the variations between the three main parameters within the data: total_bill, tip and size. The category ‘sex’ has also been passed to observe the trends in that wise. For each instance where a variable is being compared with itself, we get a kernel density estimate, or a histogram if specified. The other comparisons are made via scatterplots.

From the pairplot, we can quickly infer that the tips do not necessarily increase with increasing party size, considering that the largest tip is within the party size of 3. This inference can be found by observing graphs 2,3, and 3,2 (row,column).

You may wonder, could we also find the population size within a dataset by category? Well, countplot is very useful for that. It is common to see such kinds of plots within a document like the US census report. It basically displays a bar chart, with the height corresponding to the population of a category within the dataset.

In []: ​ sns.countplot(x ='sex', data = tips_dataset)

Out[]: <matplotlib.axes._subplots.AxesSubplot at 0x28d41990978>

To validate this, we can use the pandas groupby method along with count. Hope you can recall these methods!

In []: import pandas as pd   #importing pandas to use groupby()

  ​ tips_dataset.groupby('sex').count()

Out[]:

total_bill

tip

smoker

day

time

size

sex

Male

157

157

157

157

157

157

Female

87

87

87

87

87

87

As expected, notice how the male and female counts of 157 and 87 respectively correspond with the countplot above.

These and many more are data visualization capabilities of Seaborn. For now, these are some basic examples to get you started; you may visit the seaborn official documentation gallery to explore more plots styles and options via this link: https://seaborn.pydata.org/examples/index.html

Pandas

Well, it's our friendly pandas again. The library also has some highly functional visualization capabilities. It is quite intuitive at the time to use these built-in visualization options while working with pandas, unless something more specialized is required.

First, we import a few familiar libraries:

In []: # importing all necessary libraries

  ​ import numpy as np

  ​ import pandas as pd

  ​ import matplotlib.pyplot as plt

  ​ %matplotlib inline

  ​ import seaborn as sns

You may wonder why all the other libraries apart from Pandas are imported. Well, your outputs will look much better with these libraries synchronized. Pandas plots using the matplotlib library functionality -- even though it doesn’t directly call it, and the seaborn library makes the graphs/plots look better.

Let us work with a different dataset. We can create our own data frame and call plots off it.

We will create a data frame from a uniform distribution.

In []: # let us create our dictionary

  ​ d = {'A':np.random.rand(5),

      ​      'B':np.random.rand(5),

      ​      'C':np.random.rand(5),

      ​      'D':np.random.rand(5)}

  ​ # now creating a data frame

  ​ df = pd.DataFrame(d)

  ​ df

Out[]:

A

B

C

D

0

0.982520

0.469717

0.973735

0.397019

1

0.602272

0.148608

0.433559

0.929647

2

0.566168

0.737165

0.040840

0.435978

3

0.632309

0.772419

0.341389

0.603980

4

0.949631

0.906318

0.895018

0.679825

With our data frame, we can now observe trends. To create a histogram plot using pandas, use the hist() function. Also, you can pass some matplotlib arguments like ‘bins’

In []: df[['A']].hist(bins=30)

Out[]: array([[<matplotlib.axes._subplots.AxesSubplot object at 0x0000028D432E50B8>]],

      dtype=object)

We can do an area plot of the values as well, which is essentially a line graph of the values with the area underneath shaded:

In []: df.plot.area()

Out[]: <matplotlib.axes._subplots.AxesSubplot at 0x28d433b2438>

The transparency settings of this graph can be set with the argument ‘alpha = value’.

We can also do a bar plot which can categorize our data based off of our row_index.

In []: df.plot.bar()

Out[]: <matplotlib.axes._subplots.AxesSubplot at 0x28d435fe828>

See, our x-axis has the row index, and the y-axis shows the value of in each column per index.

This kind of plot can be useful for things like, sales trends per month (with sales as values and months as row_index), school attendance per day, etc. Our current plots might not be too informative since we are using random data, however, an actual data set would reveal more details.

If you prefer, the bar plots can be stacked to give better visualization:

In []: df.plot.bar(stacked = True, alpha = 0.8)

Out[]: <matplotlib.axes._subplots.AxesSubplot at 0x28d43a6e048>

This kind of plot gives us an idea of the total values per category, as well as the percentages that account for that total. We can still observe that value in column ‘A’ contributes the most in category ‘0’, followed by ‘C’, and so forth.

Line plot:

In []: df.plot.line(y =['B','C'])

Out[]: <matplotlib.axes._subplots.AxesSubplot at 0x28d43b6fc88>

The line plot takes positional arguments of the x and y-axis. In this case, the y-axis has been specified. Other specifications like: line width ‘lw’, figsize, etc. can be included as well.

We can also make a scatterplot, box plots and a few other plots that can be useful for interpreting data. Depending on your choice, and proficiency with these plotting techniques, you will be able to master data and the information it contains.

Go ahead and check these useful links for extra information on plotting with pandas:

https://towardsdatascience.com/introduction-to-data-visualization-in-python-89a54c97fbed

https://pandas.pydata.org/pandas-docs/stable/user_guide/visualization.html

With these data visualization options, you may start to test your skills by displaying data in a whole range of formats. It is quite obvious that a good knowledge of statistical methods would be very useful for excelling as a data scientist, since data science mostly deals with statistical data. While statistics can be intimidating without graphical aides, your own approach will be better as you now have the full potential of matplotlib, seaborn and pandas to visualize your lessons. For interactive visualization options (not covered here), you can check out ‘plotly and cufflinks’ libraries from this link: https://plot.ly/ipython-notebooks/cufflinks/

In the next section, however, we give a brief introduction into the world and concepts of machine learning.

Chapter 4

Machine Learning with Python

So, what is machine learning?

Machine learning is an advanced form of data analysis and computation that utilizes the exceptional processing speed and pattern recognition techniques of computers for finding and learning new trends in data. Quite long-winded huh?

Simply put, it is an artificial-intelligence-inspired technique of programming that allows computers to improve their learning capabilities through the data they are fed, or can access. This is much like the way human beings develop through life.

While Machine learning is not a new concept or technology -- the term was in fact first introduced at IBM in 1959 by American gaming and AI expert, Arthur Samuel – it has recently gained a lot of interests in terms of research and applications across various fields. This is mostly due to the improvement in processing capabilities of modern generation computers, along with portable, high-level, and fast programming languages that support advanced pattern recognition. The concept behind the technique is consistently being improved and tested, and it will be a key player in the bigger technology revolution of the future.

Okay, machine learning is cool. How is it related to data science?

Imagine this: You are a football player (or soccer in some countries, good grief!), and you play defense. You have tried marking the opposing team’s winger three times, and at each encounter, he did a step-over and dribbled to the left. Now, your team is defending, and that striker takes on your co-defender; you are close enough to shout instructions at him/her, what do you say?

Mark to the left!!!

Right? Unfortunately, he dribbles to the right and scores. Everyone blames you.

Okay, that story has a bad ending, but the point is that you made a prediction based on what you have learned about that striker during your previous encounters.

What if, before the game, you had watched year-long footage of that striker’s dribbling techniques and tricks? They would have a very low chance of getting past you, right? How about 10 years long footage (which you could binge watch like The Flash!)? No chance, at all!

This is where the capability of machine learning applies to data science. Computers, while not as smart as humans, are exceptionally great at finding trends (of course they are, computer operations are all about pattern recognition via one’s and zero’s). Now, imagine that machine having access to a large database, like all the tweets on twitter for the past 5 years. It would learn quite a lot about pop-culture, that’s for sure (Financial stock predictions, fraud detection, etc.). These are the most popular applications of machine learning, and the technique is based off extracting information from data; data science!

It is thus important for any current or aspiring data scientist to join the growing machine learning community, and contribute a quota to improving the technology. There are a few programming tools that are optimized for machine learning, and python is one of those.

Python and machine learning

As with most applications of python, there is a library for machine learning and it’s called Scikit Learn. The Scikit learn package is most likely included in your distribution of anaconda, and you can go ahead and verify with conda lis t at the anaconda prompt. You can check out Scikit-learn’s official documentation via this link: https://scikit-learn.org/stable/

Types of machine learning

There are currently four generalized categories of machine learning, and this knowledge is important depending on the application in mind. We have: The supervised, semi-supervised, un-supervised and Reinforcement machine learning.

In supervised learning, the desired output is known by the trainer (you, or whoever is behind the keyboard). The machine is trained using labeled inputs which it associates with corresponding outputs. Through this, the machine develops a predictive model for linking those inputs with specific outputs over a period of iterative learning. It is not so different from the way we learn in a classroom, with a teacher available to correct mistakes. This is the easiest, but usually expensive approach to machine learning.

For unsupervised learning, there is no specific goal in mind. The trainer at times does not know the right answer, and the computer only finds interesting trends in the data based on a training algorithm (usually a clustering technique). This is similar to the informal learning process in humans, where we learn based on our interaction with our environment.

The semi-supervised approach is just a scaled-down version of the supervised, which is useful in the absence of a complete labeled training data set. In this case, the machine has to make some approximations to compensate for the unlabeled data. It is cheaper than supervised learning, but slower and relatively less efficient.

Finally, the reinforcement learning technique is a trial-and-error approach based on the points-reward system in gaming (It is actually used in new gaming engines for creating competitive bosses). Here, the goal is to find the best possible route to achieve a goal. This includes using the minimum resources, while maximizing time. It is also very useful in modern robotics. Here, the machine learns from its experience while interacting with the environment.

All of these applications of machine learning are possible, and ready for exploring via Scikit learn with python. All you need is a good resource, time, dedication and all the knowledge of basic data analysis and visualization from the previous sections of this book; this is because data cleaning and preparation is a big part of machine learning.

Machine learning, like other elements of data science requires a good background in statistical analysis. Things like regression analysis i.e. linear and logistic regression, K-means clustering, K-NN (nearest neighbor), etc. Here is a link to download a good, and free book that can introduce you to some of these statistical concepts:http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Seventh%20Printing.pdf

Conclusion

We hope you have learned a ton from this book. The mastery of a skill is not just in knowing, but continuous practice, wherein lies uniqueness of skill and competence. With time, you will keep pursuing that never-ending expanse of knowledge that is data science. At least, you are no longer a novice!