Reproducible Data Science with Pachyderm

Learn how to build version-controlled, end-to-end data pipelines using Pachyderm 2.0

Svetlana Karslioglu

BIRMINGHAM—MUMBAI

Reproducible Data Science with Pachyderm

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Publishing Product Manager: Reshma Raman

Senior Editor: David Sugarman

Content Development Editor: Nathanya Dias

Technical Editor: Rahul Limbachiya

Copy Editor: Safis Editing

Project Coordinator: Aparna Ravikumar Nair

Proofreader: Safis Editing

Indexer: Pratik Shirodkar

Production Designer: Sinhayna Bais

Marketing Coordinator: Shifa Ansari

First published: March 2022

Production reference: 1070222

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80107-448-3

www.packt.com

Contributors

About the author

Svetlana Karslioglu is a seasoned documentation professional with over 10 years of experience in top Silicon Valley companies. During her tenure at Pachyderm, she authored much of the open source documentation for Pachyderm and was also in charge of the documentation infrastructure. Throughout her career, she has spoken at local conferences and given talks advocating for open infrastructure and unbiased research in artificial intelligence. When Svetlana is not busy writing books, she spends time with her three children and her husband, Murat.

This book is dedicated to my husband, Murat, who supported me through each step of the process, tirelessly watched our three kids, and cooked breakfasts, lunches, and dinners while I was busy writing this book. My husband gave me the most valuable gift of all – time to fulfill my dreams. For this, I thank you, Muratjim.

About the reviewers

Soumo Chakraborty is a data architect at Kyndryl – Data & AI team. He helps clients continually succeed by using big data and AI services. He started 15 years ago as a trainee desktop engineer and life has been about continuous learning for him – leading to his current role. He is very passionate about learning new skills around the cloud, MLOps, AIOps, and data engineering, and putting them into business use cases. Soumo loves traveling and dreams of touring the world. During his leisure time, he and his 6-year-old watch YouTube videos of the Amazon rainforest and write tech blogs.

I dedicate this effort to my parents, who taught me to keep moving, my spouse, Richa, for teaching me to challenge new heights, Manoj, a mentor for life, and to all my well-wishers.

Manoj Palaniswamy is a senior technical staff member at Kyndryl and plays a technical leadership role within the Application, Data and AI practice. He has been working in the IT industry for more than 17 years and his domain expertise includes enterprise AI strategy, data management, AIOps, MLOps, AI platform engineering, and hybrid cloud IT infrastructure and analytics. He has led many cross-cultural technical teams across the globe on complex technical projects and closely works with clients to translate their business requirements into technical solutions. Manoj holds two patents in the area of machine learning and workload optimization on VMs.

Hendrik Vincent Koops is a senior data scientist at RTL Netherlands, working on AI multimedia projects. He received a B.A. degree in audio and sound design with honors in 2008 and an M.A. degree in music composition in 2008, both at the HKU University of the Arts Utrecht. He then received B.S. (2012) and M.S. (2014) degrees in artificial intelligence at Utrecht University. After completing research at the Department of Electrical and Computer Engineering at Carnegie Mellon University, he received a Ph.D. degree in computer science from Utrecht University in 2019. At RTL Netherlands, he is responsible for developing scalable audio-visual machine learning solutions to make video content more discoverable, searchable, and valuable.

Table of Contents

Preface

Section 1: Introduction to Pachyderm and Reproducible Data Science

Chapter 1: The Problem of Data Reproducibility

Why is reproducibility important?

What is a model?

The main principles of reproducibility

The reproducibility crisis in science

Data fishing

Better reproducibility in science research guidelines

Common practices to improve reproducibility

Demystifying MLOps

Types of data science platforms

End-to-end platforms

Pluggable solutions

Data ingestion tools

Data transformation tools

Model serving tools

Data monitoring tools

Putting it all together

Explaining ethical AI

Trustworthy AI

Summary

Further reading

Chapter 2: Pachyderm Basics

Reviewing Pachyderm architecture

Why can't I use Git for my data pipelines?

Pachyderm architecture diagram

Kubernetes

Helm

Pachyderm internals

Other components

Container runtimes

Learning about version control primitives

Repository

Branch

Commit

Discovering pipeline elements

Types of pipelines

Datum

Summary

Further reading

Chapter 3: Pachyderm Pipeline Specification

Pipeline specification overview

Understanding inputs

pfs

Exploring informational parameters

name

description

metadata

Exploring transformation

image

stdin

err_cmd

err_stdin

env

secrets

image_pull_secrets

accept_return_code

debug

user

working_dir

dockerfile

Optimizing your pipeline

parallelism_spec

reprocess_spec

cache_size

max_queue_size

chunk_spec

resource_limits

resource_requests

sidecar_resource_limits

scheduling_spec

job_timeout

datum_timeout

datum_tries

Exploring service parameters

enable_stats

pod_patch

Exploring output parameters

Summary

Further reading

Section 2: Getting Started with Pachyderm

Chapter 4: Installing Pachyderm Locally

Technical requirements

Installing the required tools

Installing Homebrew (macOS only)

Installing Windows Subsystem for Linux (for Windows only)

Installing the Kubernetes command-line tool

Installing Helm v3

Installing minikube

Installing Docker Desktop

Installing Docker Desktop for macOS

Installing the Pachyderm command-line interface

Enabling autocompletion for Pachyderm

Enabling Pachyderm autocompletion for bash

Enabling Pachyderm autocompletion for zsh

Preparing the Kubernetes environment

Enabling Kubernetes on Docker Desktop

Enabling Kubernetes using minikube

Deploying Pachyderm

Accessing the Pachyderm Console

Deleting an existing Pachyderm deployment

Summary

Further reading

Chapter 5: Installing Pachyderm on a Cloud Platform

Technical requirements

Installing the required tools

Installing the AWS Command Line Interface to manage AWS

Installing the AWS IAM authenticator for Kubernetes

Installing eksctl to manage Amazon EKS

Installing the Google Cloud SDK to manage Google Cloud

Installing the Azure CLI to manage Microsoft Azure

Deploying Pachyderm on Amazon EKS

Preparing an Amazon EKS cluster to run Pachyderm

Creating an S3 object storage bucket

Deploying the cluster

Deleting a Pachyderm deployment on Amazon EKS

Deploying Pachyderm on GKE

Preparing a GKE cluster to run Pachyderm

Creating a Google Cloud object storage bucket

Deploying the cluster

Deleting a Pachyderm deployment on GKE

Deploying Pachyderm on Microsoft AKS

Preparing an AKS cluster to run Pachyderm

Creating an Azure storage container

Deploying the cluster

Deleting a Pachyderm deployment on AKS

Accessing the Pachyderm console

Summary

Further reading

Chapter 6: Creating Your First Pipeline

Technical requirements

Pipeline overview

Creating a repository

Creating a pipeline specification

Viewing the pipeline result

Adding another pipeline step

Cleaning up

Summary

Further reading

Chapter 7: Pachyderm Operations

Technical requirements

Downloading the source files

Reviewing the standard Pachyderm workflow

Executing data operations

Uploading data to Pachyderm

About data lineage

Exploring data lineage

Mounting a Pachyderm repository to a local filesystem

Executing pipeline operations

Updating your pipeline specification

Updating your code

Running maintenance operations

Troubleshooting your pipeline

Upgrading your Pachyderm cluster

Cleaning up

Summary

Further reading

Chapter 8: Creating an End-to-End Machine Learning Workflow

Technical requirements

Adjusting virtual machine parameters

NLP example overview

Introduction to NLP

Learning the NLP phases

Reviewing the NLP example

Creating repositories and pipelines

Creating the data cleaning pipeline

Creating the POS tagging pipeline

Creating an NER pipeline

Retraining an NER model

Creating the retrain pipeline

Deploying the retrained pipeline

Cleaning up

Summary

Further reading

Chapter 9: Distributed Hyperparameter Tuning with Pachyderm

Technical requirements

Reviewing hyperparameter tuning techniques and strategies

Grid search

Random search

Bayesian optimization

Regression evaluation metrics

Creating a hyperparameter tuning pipeline in Pachyderm

Example overview

Creating an exploratory analysis pipeline

Creating a data cleaning pipeline

Creating a pipeline that removes outliers

Creating a training pipeline

Creating an evaluation pipeline

Cleaning up

Summary

Further reading

Section 3: Pachyderm Clients and Tools

Chapter 10: Pachyderm Language Clients

Technical requirements

Downloading the source files

Using the Pachyderm Go client

Installing Go on your computer

Configuring $GOPATH

Cloning the Pachyderm source repository

Connecting to Pachyderm with the Go client

Creating a repository with the Go client

Putting data into a Pachyderm repository with the Go client

Creating pipelines with the Go client

Cleaning up the cluster with the Go client

Using the Pachyderm Python client

Installing the Pachyderm Python client

Connecting to your Pachyderm cluster with the Python client

Creating a Pachyderm repository with the Python client

Putting data into a Pachyderm repository with the Python client

Creating pipelines with the Pachyderm Python client

Cleaning up the cluster with the Python client

Summary

Further reading

Chapter 11: Using Pachyderm Notebooks

Technical requirements

Downloading the source files

Enabling Pachyderm Notebooks in Pachyderm Hub

Create a workspace

Connect to your Pachyderm Hub workspace with pachctl

Connect to a Pachyderm notebook

Running basic Pachyderm operations in Pachyderm Notebooks

Using the integrated terminal

Using Pachyderm Notebooks

Creating and running an example pipeline in Pachyderm Notebooks

Pipeline methodology

Creating the pipelines

Summary

Further reading

Other Books You May Enjoy

Preface

Pachyderm is a distributed version control platform for building end-to-end data science workflows. Since its creation in 2016, Pachyderm has become a go-to solution for large and small organizations. The core functionality of Pachyderm is open source and has a vivid community of engineers around it. This book walks you through basic and advanced examples of Pachyderm usage. This book will help you get started quickly and integrate a reliable data science solution into your infrastructure.

Reproducible Data Science with Pachyderm provides a clear overview of Pachyderm, as well as instructions on how to install and run Pachyderm in the cloud, and how to use the Pachyderm Software-as-a-Service (SaaS) version – Pachyderm Hub. This book has practical examples of data science technics running on a Pachyderm cluster.

Who this book is for

This book is for new and more experienced data scientists and machine learning engineers who want to build scalable infrastructures for their data science projects. Basic knowledge of Python programming and Kubernetes will be beneficial. Familiarity with GoLang is nice to have.

What this book covers

Chapter 1, The Problem of Data Reproducibility, discusses the problem of reproducibility in modern science and data science and how it aligns with the Pachyderm mission.

Chapter 2, Pachyderm Basics, describes basic Pachyderm concepts and primitives.

Chapter 3, Pachyderm Pipeline Specification, provides a detailed overview of the Pachyderm specification file, the main configuration file of Pachyderm pipelines.

Chapter 4, Installing Pachyderm Locally, walks you through the process of installing Pachyderm locally on your computer.

Chapter 5, Installing Pachyderm on a Cloud Platform, describes how to install Pachyderm on three major cloud platforms: Amazon Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE), and Microsoft Azure Kubernetes Service (AKS).

Chapter 6, Creating Your First Pipeline, covers how to create a simple pipeline that processes images.

Chapter 7, Pachyderm Operations, looks at the most often used operations.

Chapter 8, Creating an End-to-End Machine Learning Workflow, shows how to deploy an end-to-end ML workflow on an example Natural Language Processing (NLP) pipeline.

Chapter 9, Distributed Hyperparameter Tuning with Pachyderm, looks at performing distributed hyperparameter tuning with a Named-Entity Recognition (NER) pipeline.

Chapter 10, Pachyderm Language Clients, walks you through the most common examples of using Pachyderm Python and Golang clients.

Chapter 11, Using Pachyderm Notebooks, discusses the Pachyderm Hub, Pachyderm's Software-as-a-Service (SaaS) platform, and you will learn about Pachyderm Notebooks, an Integrated Development Environment (IDE) for data scientists.

To get the most out of this book

You will need to have the latest Pachyderm version installed on your computer. All operations were tested using Pachyderm 2.0 on macOS. However, they should work with future version releases too. If you are on Windows, all operations must be performed in Windows Subsystem for Linux (WSL).

You will need to request an enterprise version of Pachyderm to use the Pachyderm Console. Pachyderm provides a free trial license for first-time users. However, most examples will work without an enterprise license. To test Pachyderm Notebooks in Chapter 11, Using Pachyderm Notebooks, you will need to create a Pachyderm Hub account.

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book's GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Reproducible-Data-Science-with-Pachyderm. If there's an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781801074483_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "Mount the downloaded WebStorm-10*.dmg disk image file as another disk in your system."

A block of code is set as follows:

image = displacy.render(textfile, style='dep', options={"compact": True, "distance": 70})

f = open('/pfs/out/pos-tag-dependency.svg', "w")

f.write(image)

f.close()

Any command-line input or output is written as follows:

$ minikube start

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: "Select System info from the Administration panel."

Tips or Important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at customercare@packtpub.com and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at copyright@packt.com with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you've read Reproducible Data Science with Pachyderm, we'd love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we're delivering excellent quality content.

Section 1: Introduction to Pachyderm and Reproducible Data Science

This section introduces the basics of Pachyderm, as well as describing the importance of data reproducibility for an enterprise-level data science platform. You will learn what the main pillars of the Pachyderm solution are, including repositories, datums, jobs, and the most important of them all – the pipeline. The chapter also briefly talks about the ethics of AI in terms of reproducibility.

This section comprises the following chapters:

Chapter 1, The Problem of Data Reproducibility

Chapter 2, Pachyderm Basics

Chapter 3, Pachyderm Pipeline Specification

Chapter 1: The Problem of Data Reproducibility

Today, machine learning algorithms are used everywhere. They are integrated into our day-to-day lives, and we use them without noticing. While we are rushing to work, planning a vacation, or visiting a doctor's office, the models are at work, at times making important decisions about us. If we are unsure what the model is doing and how it makes decisions, how can we be sure that its decisions are fair and just? Pachyderm profoundly cares about the reproducibility of data science experiments and puts data lineage, reproducibility, and version control at its core. But before we proceed, let's discuss why reproducibility is so important.

This chapter explains the concepts of reproducibility, ethical Artificial Intelligence (AI), and Machine Learning Operations (MLOps), as well as providing an overview of the existing data science platforms and how they compare to Pachyderm.

In this chapter, we're going to cover the following main topics:

Why is reproducibility important?

The reproducibility crisis in science

Demystifying MLOps

Types of data science platforms

Explaining ethical AI

Why is reproducibility important?

First of all, let's define AI, ML, and data science.

Data science is a field of study that involves collecting and preparing large amounts of data to extract knowledge and produce insights.

AI is more of an umbrella term for technology that enables machines to mimic the behavior of human beings. Machine learning is a subset of AI that is based on the idea that an algorithm can learn based on past experiences.

Now, let's define reproducibility. A data science experiment is considered reproducible if other data scientists can repeat it with a comparable outcome on a similar dataset and problem. And although reproducibility has been a pillar of scientific research for decades, it has only recently become an important topic in the data science scope.

Not only is a reproducible experiment more likely to be free of errors, but it also takes the experiment further and allows others to build on top of it, contributing to knowledge transfer and speeding up future discoveries.

It's not a secret that data science has become one of the hottest topics in the last 10 years. Many big tech companies have opened tens of high-paying data scientist, data engineering, and data analyst positions. With that, the demand to join the profession has been rising exponentially. According to the AI Index 2019 Annual Report published by the Stanford Institute for Human-Centered Artificial Intelligence (HAI), the number of AI papers has grown threefold in the last 20 years. You can read more about this report on the Stanford University HAI website: https://hai.stanford.edu/blog/introducing-ai-index-2019-report.

Figure 1.1 – AI publications trend, from the AI Index 2019 Annual Report (p. 5)

Almost every learning platform and university now offers a data science or AI program, and these programs never lack students. Thousands of people of all backgrounds, from software developers to CEOs, take ML classes to keep up with the rapidly growing industry.

The number of AI conferences has been steadily growing as well. Even in the pandemic world, where in-person events have become impossible, the AI community has continued to meet in a virtual format. Such flagship conferences as Neural Information Processing Systems (NeurIPS) and International Conference on Machine Learning (ICML), which typically attract more than 10,000 visitors, took place online with significant attendance.

According to some predictions, the AI market size will increase to more than $350 billion by 2025. The market grew from $12 billion to $58 billion from 2020 to 2021 alone. The Silicon Valley tech giants are fiercely battling to achieve dominance in the space, while smaller players emerge to get their share of the market. The number of AI start-ups worldwide is steadily growing, with billions being invested in them each year.

The following graph shows the growth of AI-related start-ups in recent years:

Figure 1.2 – Total private investment in AI-related start-ups worldwide, from the AI Index 2019 Annual Report (p. 88)

The total private investment in AI start-ups grew by more than 30 times in the last 10 years.

And another interesting metric from the same source is the number of AI patents published between 2015 and 2018:

Figure 1.3 – Total number of AI patents (2015-2018), from the AI Index 2019 Annual Report (p. 32)

The United States is leading in the number of published patents among other countries.

These trends boost the economy and industry but inevitably affect the quality of submitted AI papers, processes, practices, and experiments. That's why a proper process is needed to ensure the validation of data science models. The replication of experiments is an important part of a data science model's quality control.

Next, let's learn what a model is.

What is a model?

Let's define what a model is. A data science or AI model is a simplified representation of a process that also suggests possible results. Whether it is a weather-prediction algorithm or a website attendance calculator, a model provides the most probable outcome and helps us make informed decisions. When a data scientist creates a model, they need to make decisions about the critical parameters that must be included in that model because they cannot include everything. Therefore, a model is a simplified version of a process. And that's when sacrifices are made based on the data scientist's or organization's definition of success.

The following diagram demonstrates a data model:

Figure 1.4 – Data science model

Every model needs a continuous data flow to improve and perform correctly. Consider the Amazon Go stores where shoppers' behavior is analyzed by multiple cameras inside the store. The models that ensure safety in the store are trained continuously on real-life customer behavior. These models had to learn that sometimes shoppers might pick up an item and then change their mind and put it back; sometimes shoppers can drop an item on the floor, damaging the product, and so on. The Amazon Go store model is likely good because it has access to a lot of real data, and it improves over time. However, not all models have access to real data, and that's when a synthetic dataset can be used.

A synthetic dataset is a dataset that was generated artificially by a computer. The problem with synthetic data is that it is only as good as the algorithm that generated it. Often, such data misrepresents the real world. In some cases, such as when users' privacy prevents data scientists from using real data, usage of a synthetic dataset is justified; in other cases, it can lead to negative results.

IBM's Watson was an ambitious project that promised to revolutionize healthcare by promising to diagnose patients based on a provided list of symptoms in a matter of a few seconds. This invention could greatly speed up the diagnosis process. In some places on this planet, where people have no access to healthcare, a system like that could save many lives. Unfortunately, since the original promise was to replace doctors, Watson is a recommendation system that can assist in diagnosing, but nothing more than that. One of the reasons is that Watson was trained on a synthetic dataset and not on real data.

There are cases when detecting issues in a trained model can be especially difficult. Take the example of an image recognition algorithm developed by the University of Washington that was built to identify whether an image had a husky portrayed in it or a wolf. The model was seemingly working really well, predicting the correct result with almost 90% accuracy. However, when the scientists dug a bit deeper into the algorithm and data, they learned that the model was basing its predictions on the background. The majority of images with huskies had grass in the background, while the majority of images with wolves had snow in the background.

The main principles of reproducibility

How can you ensure that a data science process in your company adheres to the principles of reproducibility? Here is a list of the main principles of reproducibility:

Use open data: The data that is used for training models should not be a black box. It has to be available to other data scientists in an unmodified state.

Train the model on many examples: The information about experiments and on how many examples it was trained must be available for review.

Rigorously document the process: The process of data modifications, statistical failures, and experiment performance must be thoroughly documented so that the author and other data scientists can reproduce the experiment in the future.

Let's consider a few examples where reproducibility, collaboration, and open data principles were not part of the experiment process.

A few years ago, a group of scientists at Duke University became wildly popular because they emerged with an ambitious claim of predicting the course of lung cancer based on the data collected from patients. The medical community was very excited about the prospect of such a discovery. However, a group of other scientists in the MD Anderson Cancer Centre in Houston found severe errors in that research when they tried to reproduce the original result. They discovered mislabeling in the chemotherapy prediction model, mismatches in genes to gene-expression data, and other issues that would make correct treatment prescription based on the model calculations significantly less likely. While the flaws were eventually unraveled, it took almost 3 years and more than 2,000 working hours for the researchers to get to the bottom of the problem, which could have been easily avoided if the proper research practices were established in the first place.

Now let's look at how AI can go wrong based on a chatbot example. You might remember the infamous Microsoft chatbot called Tay. Tay was a robot who could learn from his conversations with internet users. When Tay went live, his first conversations were friendly, but overnight his language changed, and he started to post harmful, racist, and overall inappropriate responses. He learned from the users who taught him to be rude, and as the bot was designed to mirror human behavior, he did what he was created for. Why was he not racist from the very beginning, you might ask? The answer is that he was trained on clean, cherry-picked data that did not include vulgar and abusive language. But we cannot control the web and what people post, and the bot did not have any sense of morals programmed into it. This experiment raised many questions about AI ethics and how we can ensure that the AI that we build does not turn on us one day.

The new generation of chatbots is built on the recently released GPT-3 library. These chatbots are trained with neural networks that, during training, create associations that cannot be broken. These chatbots, although using a seemingly more advanced technology behind them than their predecessors, still easily might become racists and hateful depending on the data they are trained on. If a bot is trained on misogynist and hateful conversations, it will be offensive and will likely reply inappropriately.

As you can see, data science, AI, and machine learning are powerful technologies that help us solve many difficult problems, but at the same time, they can endanger their users and have devastating consequences. The data science community needs to work on devising better ways of minimizing adverse outcomes by establishing proper standards and processes to ensure the quality of data science experiments and AI software.

Now that we've seen why reproducibility is so important, let's look at what consequences it has on the scientific community and data science.

The reproducibility crisis in science

The reproducibility crisis is a problem that has been around for more than a decade. Because data science is a close discipline to science, it is important to review the issues many scientists have outlined in the past and correlate them with similar problems the data science space is facing today.

One of the most important issues is replication—the ability to reproduce the results of a scientific experiment has been one of the founding principles of good research. In other words, if an experiment can be reproduced, it is valid, and if not, it could be a one-time occurrence that does not represent real phenomena. Unfortunately, in recent years, more and more research papers in sociology, medicine, biology, and other areas of science cannot withhold retesting against an increased number of samples, even if these papers were published in well-known and trustworthy science magazines, such as Nature. This tendency could lead to public mistrust in science and AI as part of it.

As was mentioned previously, because of the popularity and growth of the AI industry, the number of AI papers has increased multiple times. Unfortunately, the quality of these papers does not grow with the number of papers itself.

Nature magazine recently conducted a survey among scientists asking them whether they feel that there is a reproducibility crisis in science. The majority of scientists agreed that false-positive results due to pressure to publish results frequently definitely exists. Researchers need sponsorship and sponsors need to see results to invest additional money in the research, which results in many published papers with declining credibility. Ultimately, the fight for grants and bureaucracy are often named as the main causes of the lack of the reproducibility process in labs.

The research papers that were questioned for integrity have the following common attributes:

No code or data were publicly shared for other researchers to attempt to replicate the results.

The scientists who attempted to replicate the results failed completely or partially to do it by following the provided instructions.

Even the papers published by Nobel laureates can sometimes be questioned due to an inability to reproduce the results. For example, in 2014, Science magazine retracted a paper published by Nobel Prize winner and immunologist Bruce Beutler. His paper was about the response to pathogens by virus-like organisms in the human genome. This paper was cited over 50 times before it was retracted.

When COVID-19 become a major topic of 2020, multiple papers were published on it. According to Retraction Watch, an online blog that tracks the scientific papers that have been called off, as of March 2021 more than 86 of them were retracted.

In 2019, more than 1,400 science papers were retracted by multiple publishers. This number is huge and has been steadily growing, compared to only 50 papers in the early 2000s. This raises awareness of a so-called reproducibility crisis in science. While not all papers are retracted for that reason, oftentimes it happens because of that.

Data fishing

Data fishing or data dredging is a method of achieving a statistically significant result of an experiment by running the computation multiple times before the desired result is achieved and only reporting these results and ignoring the inconvenient results. Sometimes, scientists unintentionally dredge the data to achieve the result they think is most probable and confirm their hypothesis. A more sinister plan can take place too—a scientist might be intentionally hacking the result of the experiment to achieve a predefined conclusion.

An example of such a misuse of data analysis would be if you decided to prove that there is a correlation between banana consumption and an increased level of IQ in children of age 10 and older. This is a completely made-up example, but say you wanted to establish this connection. You would need to get information about IQ level and banana consumption of a big enough sample of children – let's say 5,000.

Then, you would run tests, such as: do kids who eat bananas and exercise have a higher IQ level than the ones who only exercise? Do kids who watch TV and eat bananas have a higher level of IQ compared to the ones who do not? After conducting these tests enough times, you most likely would get some kind of correlation. However, this result would not be significant, and using the data dredging technique is considered extremely unethical by the scientific community. In data science specifically, similar problems are being seen.

Without conducting a full investigation, detecting data dredging might be difficult. Possible factors to look for include the following:

Was the research conducted by a reputable institution or group of scientists?

What does other research in similar areas suggest?

Is financial interest involved?

Is the claim sensational?

Without a proper process, data dredging and unreliable researchers will continue to be published. Recently, Nature magazine surveyed around 1,500 researchers from different areas of science and more than 50% of respondents outlined that they have tried and failed to reproduce the results of research in the past. Even more shockingly, in many cases, they failed to reproduce the results of their own experiments.

Out of all respondents, only 24% were able to successfully publish their reproduction attempts and the majority were never contacted with a request to reproduce someone else's research.

Of course, increasing the reproducibility of experiments is a costly problem and can double the time required to conduct an experiment, which many research laboratories might not be able to afford. But if it's added to the originally planned time for the research and has a proper process, it should not be as difficult or rigorous as adding it midway in the research lifecycle.

Even worse, retracting a paper after it was published can be a tedious task. Some publishers even charge researchers a significant amount of money if a paper is retracted. Such practices are truly discouraging.

All of this negatively impacts research all over the world and results in growing mistrust in science. Organizations must take steps to improve processes in their scientific departments and scientific journals must raise the bar of publishing research.

Now that we have learned about data fishing, let's review better reproducibility guidelines.

Better reproducibility in science research guidelines

The Center for Open Science (COS), a non-profit organization that focuses on supporting and promoting open-science initiatives, reproducibility, and integrity of scientific research, has published Guidelines for Transparency and Guidelines for Transparency and Openness Promotion (TOP) in Journal Policies and Practices, or the TOP Guidelines. These guidelines emphasize the importance of transparency in published research papers. Researchers can use them to justify the necessity of sharing research artifacts publicly to avoid any possible inquiries regarding the integrity of their work.

The main principles of the TOP Guidelines include the following:

Proper citation and credit to original authors: All text, code, and data artifacts that belong to other authors must be outlined in the paper and credit given as needed.

Data, methodology, and research material transparency: The authors of the paper must share the written code, methodology, and research materials in a publicly accessible location with instructions on how to access and use them.

Design and analysis transparency: The authors should be transparent about the methodology as much as possible, although this might vary by industry. At a minimum, they must disclose the standards that have been applied during the research.

Preregistrations of the research and analysis plans: Even if research does not get published, preregistration makes it more discoverable.

Reproducibility of obtained results: The authors must include sufficient details on how to reproduce the original results.

There are three levels that are applied to all these metrics:

Not implemented—information is not included in the report

Level 1—available upon request

Level 2—access before publication

Level 3—verification before publication

Level 3 is the highest level of transparency that a metric can achieve. Having this level of transparency justifies the quality of submitted research. COS applies the TOP factor to rate a journal's efforts to ensure transparency and ultimately the quality of the published research.

Apart from data and code reproducibility, often the environment and software used during the research play a big role. New technologies, such as containers and virtual and cloud environments make it easy to achieve uniformity in conducted research. Of course, if we consider biochemistry or other industries that require more precise lab conditions, achieving uniformity might be even more complex.

Now let's learn about common practices that help improve reproducibility.

Common practices to improve reproducibility

Thanks to the work of reproducibility advocates and the problem being widely discussed in scientific communities in recent years, some positive tendencies in increasing reproducibility seem to be emerging. These practices include the following:

Request a colleague to reproduce your work.

Develop extensive documentation.

Standardize research methodology.

Preregister your research before publication to avoid data cherry-picking.

There are scientific groups that make it their mission to reproduce and notify researchers about mistakes in their papers. Their typical process is to try to reproduce the result of a paper and write a letter to the researchers or lab to request a correction or retraction. Some researchers willingly collaborate and correct the mistakes in the paper, but in other cases, it is unclear and difficult. One such group has identified the following problems in the 25 papers that they analyzed:

Lack of process or point of contact regarding to whom they should address feedback on a paper. Scientific journals do not provide a clear statement on whether feedback can be addressed to the chief editor or whether there is a feedback submission form of some sort.

Scientific journal editors accept and act on submissions unwillingly. In some cases, it might take up to a year to publish a warning on a paper that has received critical feedback, even if it was provided by a reputable institution.

Some publishers expect you to pay if you want to publish a correction letter and delay retractions.

Raw data is not always available publicly. In many cases, publishers did not have a unified process around a shared location for the raw data used in the research. If you have to directly contact an author, you might not be able to get the requested information and it might significantly delay the process. Moreover, they can simply deny such a request.

The lack of a standard in submitting corrections and research paper retractions contributes to the overall reproducibility crisis and knowledge sharing. The papers that used data dredging and other techniques to manipulate the results will become a source of information for future researchers, contributing to the overall misinformation and chaos. Researchers, publishers, and editors should work together on establishing unified post-publication review guidelines that encourage other scientists to participate in testing and providing feedback.

We've learned how reproducibility affects the quality of research. Now, let's review how organizations can establish a process to ensure their data science experiments adhere to best industry practices to ensure high standards.

Demystifying MLOps

This section defines Machine Learning Operations (MLOps) and describes why it is crucial to establish a reliable MLOps process within your data science department.

In many organizations, data science departments have been created fairly recently, in the last few years. The profession of data scientist is fairly new as well. Therefore, many of these departments have to find a way to integrate into the existing corporate process and devise ways to ensure the reliability and scalability of data science deliverables.

In many cases, the burden of building a suitable infrastructure falls on the shoulders of the data scientists themselves, who often are not as familiar with the latest infrastructure trends. Another problem is how to make it all work for different languages, platforms, and environments. In the end, data scientists spend more time on building the infrastructure than on working on the model itself. This is where the new discipline has emerged to help bridge the gap between data science and infra.

MLOps is a lifecycle process that identifies the stages of machine learning operations, ensuring the reliability of the data science process. MLOps is a set of practices that define the machine learning development process. Although the term was coined fairly recently, most data scientists agree that a successful MLOps process should adhere to the following principles:

Collaboration: This principle implies that everything that goes into developing an ML model must be shared among data scientists to preserve knowledge.

Reproducibility: This principle implies that not only the code but datasets, metadata, and parameters should be versioned and reproducible for all production models.

Continuity: This principle implies that a lifecycle of a model is a continuous process that means repetition of the lifecycle stages and improvement of the model with each iteration.

Testability: This principle implies that the organization implements ML testing and monitoring practices to ensure the model's quality.

Before we dive into the MLOps process stages, let's take a look at more established software development practices. DevOps is a software development practice that is used in many enterprise-level software projects. A typical DevOps lifecycle includes the following stages that continuously repeat, ensuring product improvement:

Planning: In this stage, the overall vision for the software is developed, and a more detailed design is devised.

Development: In this stage, the code is written, and the planned functionality is implemented. The code is shared through version control systems, such as Git, which ensures collaboration between software developers.

Testing: In this stage, the developed code is tested for defects through an automated or manual process.

Deployment: In this stage, the code is released to production servers, and the users have a chance to test it and provide feedback.

Monitoring: In this stage, the DevOps engineers focus on software performance and causes of outages, identifying possible areas of improvement.

Operations: This stage ensures the automated release of software updates.

The following diagram illustrates the DevOps lifecycle:

Figure 1.5 – DevOps Lifecycle

All these phases are continuously repeated, enabling communication between departments and a customer feedback loop. This practice has brought enterprises such benefits as a faster development cycle, better products, and continuous innovation. Better teamwork enabled by the close relationships between departments is one of the key factors that make this process efficient.

Data scientists deserve a process that brings the same level of reliability. One of the biggest problems of enterprise data science is that very few machine learning models make it to production. Many companies are just starting to adopt data science, and the new departments face unprecedented challenges. Often, the teams lack an understanding of the workflows that need to be implemented in order to make enterprise-level data science work.

Another important challenge is that unlike in traditional software development, data scientists operate not only with code but also with data and parameters. Data is taken from the real world, and the code is accurately developed in the office. The only time they cross is when they are combined in a data model.

The challenges that all data science departments face include the following:

Inconsistent or totally absent data science processes

No way to track data changes and reproduce past results

Slow performance

In many enterprises, data science departments are still small and struggle to create a reliable workflow. Building such a process requires certain expertise, such as an understanding of traditional software practices, such as DevOps, mixed with an understanding of data science challenges. That is where MLOps started to emerge, combining data science with best practices of software development.

If we try to apply similar DevOps practices to data science, here is what we might see:

Design: In this phase, data scientists work on acquiring the data and designing a data pipeline, also known as an Extract, Transform, Load (ETL) pipeline. A data pipeline is a sequence of transformation steps data goes through, which ends with an output result.

Development: In this stage, data scientists work on writing the algorithmic code for the previously developed data pipeline.

Training: In this stage, the model is trained with the selected or autogenerated data. During this stage, such techniques as hyperparameter tuning can be used.

Validation: In this stage, the trained data is validated to work with the rest of the data pipeline.

Deployment: In this stage, the trained and validated model is deployed into production.

Monitoring: In this stage, the model is constantly monitored for performance and possible flaws, and feedback is delivered directly to the data scientist for further improvement.

Similar to DevOps, the stages of MLOps are constantly repeated. The following diagram shows the stages of MLOps:

Figure 1.6 – MLOps Lifecycle

As you can see, the two practices are very similar, and the latter borrows the main concepts from the former. Using MLOps in practice has brought the following advantages to enterprise-level data science:

Faster go-to-market delivery: A data science model only has value when it is successfully deployed in production. With so many companies struggling to implement a proper process in their data science departments, an MLOps solution can genuinely make a difference.

Cross-team collaboration and communication: Software-development practices applied to data science create a common ground for developers, data scientists, and IT operations to work together and speak the same language.

Reproducibility and knowledge transfer: Keeping the code, the datasets, and the history of changes plays a big role in the improvement of overall model quality and enables data scientists to learn from each other's examples, contributing to innovation and feature development.

Automation: Automating a data pipeline helps to keep the process consistent across multiple releases and speeds up the promotion of a Proof of Concept (POC) model to a production-grade pipeline.

In this section, we've learned about the important stages of the MLOps process. In the next section, we will learn more about the types of data science platforms that can help you implement MLOps in your organization.

Types of data science platforms

This section walks you through the data science platforms that are available in the open source world and on the market today and will help you understand the difference between them.

As new fields of AI and machine learning emerge, more and more engineers are working on new ways of solving data science problems, creating an infrastructure for better, faster AI adoption. Some platforms provide end-to-end capabilities for data from a data warehouse all the way to production, while others offer partial functionality and work in combination with other tools. Generally, there is no solution that fits all use cases, and certainly not every budget.

However, all of these solutions completely or partially facilitate the following stages of a data science lifecycle:

Data Engineering

Data Acquisition and Transformation

Data Training

Model Deployment

Monitoring and Improvement

The following diagram shows the types of data science tools:

Figure 1.7 – Types of data science tools

Let's take a look at the existing data science platforms that can help you to build your data science workflow at scale.

End-to-end platforms

An end-to-end data science solution should be able to provide the tooling for all the stages of the ML lifecycle listed in the previous section. However, in some use cases, the definition of the end-to-end workflow could be different and might mostly work with the ML pipelines and projects, excluding the data engineering part. Since the definition may still fluctuate, it is likely that the end-to-end tools will continue to provide different functionalities as the field evolves.

If such a platform does exist, it should bring the following benefits:

A unified user interface that eliminates the need to stitch multiple interfaces together

Collaboration for all involved individuals, including data scientists, data engineers, and IT operations

The convenience of infrastructure support being offloaded to the solution provider, which offers the team additional time to focus on data models rather than on infrastructure problems

However, you might find the following disadvantages of an end-to-end platform to be inconsistent with your organization's goals:

Portability: Such a platform would likely be proprietary, and migration to a different platform would be difficult.

Price: An end-to-end platform will likely be subscription-based, which many data science departments might not be able to afford. If GPU-based workflows are involved, the price increases even more.

Bias: When you are using a proprietary solution that offers built-in pipelines, your models are bound to inherit bias from these automated tools. The problem is that bias might be difficult to recognize and address in automated ML solutions, which could potentially have negative consequences for your business.

Now that we are aware of the advantages and disadvantages of end-to-end data science platforms, let's consider the ones that are available on the market today. Because the AI field is developing rapidly, new platforms emerge every year. We'll look into the top five such platforms.

Big tech giants, such as Microsoft, Google, and Amazon, all offer automated ML features that a lot of users might find useful. Google's AI Platform offers Kubeflow pipelines to help manage ML workflows. Amazon offers tools that assist with hyperparameter tuning and labeling. Microsoft offers Azure Machine Learning services that support GPU-based workflows and are similar to Amazon's services functionality.

However, as stated previously, all these Explainable AI (XAI) features are prone to bias and require the data science team to build additional tools that can verify model performance and reliability. For many organizations, automated ML is not the right answer. Another issue is vendor lock-in, as you will have to keep all your data in the underlying cloud storage.

The Databricks solution provides a more flexible approach as it can be deployed on any cloud. Databricks is based on Apache Spark, one of the most popular tools for AI and ML workflows and offers end-to-end ML pipeline management through a platform called MLflow. MLflow enables data scientists to track their pipeline progress from model development to deployment to production. Many users enjoy the built-in notebook interface. One disadvantage is the lack of data visualization tools, which might be added in the future.

Algorithmia is another proprietary solution that can be deployed on any cloud platform and that provides an end-to-end ML workflow with model training, deployment, versioning, and other built-in functionality. It supports batch processing and can be integrated with GitHub actions. While Algorithmia is a great tool, it has some of the traditional software developer tools built in, which some engineering teams might find redundant.

Pluggable solutions

While end-to-end platforms might sound like the right solution for your data science department, in reality, it is not always the case. Big companies often have requirements that end-to-end platforms cannot meet. These requirements might include the following:

Data security: Some companies might have privacy limitations on storing their data in the cloud. These limitations also apply to the use of automated ML features.

Pipeline outputs: Often, the final product of a pipeline is a library that is packaged and used in other projects within the organization.

Existing infrastructure constraints: Some existing infrastructure components might prevent the integration of an end-to-end platform. Some parts of the infrastructure might already exist and satisfy the user's needs.

Pluggable solutions give data infrastructure teams the flexibility to build their own solution, which also comes with the need to support it. However, most of the big companies end up doing just that.

Pluggable solutions can be divided into the following categories:

Data ingestion tools

Data transformation tools

Data serving tools

Data visualization and monitoring tools

Let's consider some of these tools, which can be combined together to build a data science solution.

Data ingestion tools

Data ingestion is the process of collecting data from all sources in your company, such as databases, social media, and other platforms, into a centralized location for further consumption by machine learning pipelines and other AI processes.

One of the most popular open source tools to ingest data is Apache NiFi, which can ingest data into Apache Kafka, an open source streaming platform. From there, data pipelines can consume the data for processing.

Among commercial cloud-hosted platforms, we can name Wavefront, which enables not only ingestion but data processing as well. Wavefront is notable for its ability to scale and support high query loads.

Data transformation tools

Data transformation is the process of running your code against the data you have. This includes training and testing your data as part of a data pipeline. The tool should be able to consume the data from a centralized location. Tools such as TensorFlow and Keras provide extended functionality for this type of operation.

Pachyderm is a data transformation and pipeline tool as well, although its main value is in version control for large datasets. Unlike other transformation tools, Pachyderm gives data scientists the freedom to define their own pipelines and supports any language and library.

If you have taken any data science classes, chances are you have used MATLAB or Octave for model training. These tools provide a great playground to start exploring machine learning. However, when it comes to production-grade data science that requires continuous training, collaboration, version control, and model productization, these tools might not be the best choice. MATLAB and Octave are mainly for numerical computing for academic purposes. Another issue with platforms such as MATLAB is that they often use proprietary languages, while tools like Pachyderm support any language, including the most popular ones in the data science community.

Model serving tools

After you train your model and it gives satisfactory results, you need to think about moving that model into production, which often is convenient to do in the form of a REST API or through a table that is ingested into a database. Depending on the language that is used in your model, serving a REST API can be as easy as using a web framework such as Flask.

However, there are more advanced tools that can that give data scientists end-to-end control over the machine learning process. One such open source tool is Seldon. Seldon converts REST API endpoints into a production microservice, where you can easily promote each version of your model from staging to production.

Another tool that provides similar functionality is KFServing. Both solutions use Kubernetes' Custom Resource Definition (CRD) to define a Deployment class for model serving.

Often, in big companies, different teams are responsible for training models and serving models, and therefore, decisions can be made based on the team's familiarity and preference for one or the other solution.

Data monitoring tools

After the model is deployed in production, data scientists need to continue to receive feedback about model performance, possible bias, and other metrics. For example, if you have an e-commerce website with a recommendation system that suggests to users what to buy with the current order based on their past choices, you need to make sure that the system is still on track with the latest fashion trends. You might not know the trends, but the feedback loop should signal a decrease in model performance when it occurs.

Often, enterprises fail to employ a good monitoring solution for ML workflows, which can have a potentially devastating outcome for your business. Seldon Alibi is one of the tools that provide model inspection functionality, which enables data scientists to monitor models running in production and identify areas of improvement. Seldon Alibi provides outlier detection, which helps to discover anomalies; drift detection, which helps monitor changes in correlation between input and output data; and adversarial detection, which exposes malicious changes in the original data inputs.

Fiddler is another popular tool that monitors a production model for integrity, bias, performance, and outlier anomalies.

Putting it all together

As you can see, there are multiple ways to create a production-grade data science solution, and one size likely will not fit all. Although end-to-end solutions provide the convenience of using one vendor, they also have multiple disadvantages and are likely equipped with inferior domain functionality compared to pluggable tools. Pluggable tools, on the other hand, require certain expertise and culture to be present in your organization, which will allow different departments, such as DevOps engineers, to collaborate with data scientists to build an optimal solution and workflow.

The next section will walk us through the ethical problems that plague modern AI applications, how they might affect your business, and what you can do about them.

Explaining ethical AI

This section describes aspects of ethical problems in AI and what organizations need to be aware of when they build artificial intelligence applications.

With AI and machine learning technologies becoming more widespread and accepted, it is easy to lose track of the data and the decision-making process origins. When an AI algorithm suggests which pair of shoes to buy based on your recent searches, it might not be a big deal. But suppose an AI algorithm is used to decide whether you qualify for a job, how likely you are to commit a crime, or whether you qualify for mortgage approval. In that case, it is essential to know how the algorithm was created, on which data it was trained, what was included in the dataset, and, more importantly, what was not. At a minimum, we need to question whether a proper process existed to validate the data used for producing the model. Not only is this the right thing to do, but it could also save your organization from undesirable legal consequences.

While AI applications bring certain advantages and improve the quality of our lives, they can make mistakes that sometimes can have adverse, and even devastating, effects on people's lives. These tendencies resulted in the emergence of ethical AI teams and ethical AI advocates in leading AI companies and big tech.

Ethical AI has been an increasingly discussed topic in the data science community over the last few years. According to the Artificial Intelligence Index Report 2019, ethics has been a steadily growing keyword in the total number of AI papers at leading AI conferences.

Figure 1.8 – Number of AI conference papers mentioning Ethics since 1970, from the AI Index 2019 Annual Report (p. 44)

Let's consider one of the most widely criticized AI technologies—facial recognition. A face recognition application can identify a person in an image or video. In recent years, this technology has become widespread and is now used in home security, authentication, and other areas. In 2018-2019, more than 1,000 newspaper articles worldwide mentioned facial recognition and data privacy. One such cloud-based facial recognition technology called Rekognition, developed by Amazon, has been used by police departments in a few states. The law enforcement departments used the software to search for suspects in a database, in a video surveillance analysis, including the feed from police body cameras. Independent research showed that the software was biased against people of color when out of 120 Members of Congress, it recognized 28 of them as potential criminals. All of them had darker skin tones. The tool performed especially poorly on identifying women of color.

The problem with this and other facial recognition technologies is that it was trained on a non-inclusive dataset that had photographs of mostly white men. Such outcomes are difficult to predict, but this is what ethical AI is trying to do. Implementing a surveillance system like that in public places would have negatively affected thousands of people. Advances in AI made facial recognition technology that requires little to no human involvement in subject identification. This raises the problem of total control and privacy. While a system like that could help identify criminals, possibly prevent crimes, and make our society safer, it needs to be thoroughly audited for potential errors and protected from misuse. With great power comes great responsibility.

Another interesting example is in using Natural Language Processing (NLP) applications. NLP is an ML technology that enables machines to automatically interpret and translate texts written in one human language to another. In recent years, NLP applications have seen major advances. Tools such as Google Translate solved a problem that was unsolvable even 20 years ago. NLP breaks down a sentence into chunks and tries to make connections between those chunks to provide a meaningful interpretation. NLP applications deal not only with translations but can also summarize what is written in a lengthy research paper or convert text to speech.

But these applications can make mistakes as well. One example was discovered in translations from Turkish to English. In the Turkish language, there is only the personal pronoun o, which can mean either she/her or he/his. It was discovered that Google Translate was discriminating based on gender, diminishing women's roles based on common stereotypes. For example, it would translate She is a secretary and He is a doctor, although in Turkish, both of these sentences could be written about a man or a woman.

From these examples, you can see that bias is one of the biggest problems of AI applications. A biased dataset is a dataset that does not include enough samples of a studied phenomenon to output an objective result, like in the facial-recognition example above, which did not have enough representatives of people of color to make a correct prediction.

While many companies are becoming aware of the adverse effects and risks of bias in datasets, few of them are taking steps to mitigate the possible negative consequences. According to the Artificial Intelligence Index Report 2019, only 13% of organizations that responded were working toward improving the equity and fairness of the datasets used:

Figure 1.9 – Types of organizations taking steps to mitigate the risks of AI, from the AI Index 2019 Annual Report (p. 102)

Another aspect of bias is financial inequality. It's not a secret that people from less economically advantageous backgrounds have harder times getting credit deals than those from a more fortunate background. Credit reports are known to have errors that cause higher borrowing rates.

Companies whose business is creating customer profiles, or personalization, go even further collecting intimate information about users and their behavior from public records, credit card transactions, sweepstakes, and other sources. These reports can be sold to marketers and even law enforcement organizations. Individuals are categorized according to their sex, age, marital status, wealth, medical conditions, and other factors. Sometimes these reports have outdated information about things such as criminal records. There was a case when an old lady could not get into a senior living house because of an arrest. However, though she was arrested, it was a case of domestic violence from her partner and she was never prosecuted. She was able to correct her police records, but not the report created by a profiling company. Correcting mistakes in the reports created by these companies is extremely difficult and they can affect people's lives for decades.

Sometimes, people get flagged because of a misidentified profile. Imagine that you are applying for a job and are denied because you have been prosecuted for theft or burglary in the past. This could come as a shock and might not make any sense, but there are cases like that with people who have common names. To clear a mistake like that you need the intervention of a person who wants to spend time correcting such mistakes for you. But do you meet people like that often?

With machine learning now being used in customer profiling, many data privacy advocates question the methods being used in these algorithms. Because these algorithms learn from past experiences, according to them anything you've done in the past, you are likely to repeat in the future. According to these algorithms, criminals will commit more crimes and the poor will get poorer. There is no room for mistakes in their reality. This means that people with prior convictions will likely get arrested again, which gives law enforcement a base for discrimination. The opposite is also true: those with a perfect record, from a better neighborhood, are not likely to commit a crime. This does not sound fair.

The problem with recidivism models is that most of them are proprietary black boxes. A black-box model is an end-to-end model that is created by an algorithm directly from the provided data and even a data scientist cannot explain how it makes decisions. When a machine learning algorithm evolves over time, since AI algorithms learn similarly to humans, they learn the same biases as us.

Figure 1.10 – Black-box model

Let's move on to the next section!

Trustworthy AI

While a few years ago, ethical AI was something only a few groups of independent advocates and academics were working on, today more and more big tech companies have established ethical AI departments to protect the companies from reputational and legal risks.

Establishing standards for trustworthy AI models is an ambitious task and one size does not fit all. However, the following principles apply to most cases:

Create an ethical AI committee that works on discussing the AI-associated risks in alignment with the overall company strategy.

Raise awareness of the dangers of non-transparent machine learning algorithms and the potential risks they pose to society and your organization.

Create a process of identifying, communicating, and evaluating biased models, and privacy concerns. For example, in healthcare, protecting patient personal information is vitally important. Create ownership around ethical risk in the product management department.

Establish a process of notifying users about how their data will be used, explaining the risk of bias and other concepts in plain English. The earlier the user becomes aware of the implications of using your application, the less legal risk this will pose in the future.

Build a culture around praising efforts to promote ethical programs and initiatives to motivate employees to contribute to those efforts. Engage employees from different departments, including engineering, data science, product management, and others, to contribute to those efforts.

According to the Artificial Intelligence Index Report 2019, the top AI ethics challenges include fairness, interpretability and explainability, and transparency.

The following figure shows a more complete list of challenges present in the ethical AI space:

Figure 1.11 – Ethical AI challenges, from the AI Index 2019 Annual Report (p. 149)

The following is a list of issues that non-transparent machine learning algorithms may cause:

Disproportional spread of economic and financial opportunities, including credit discrimination and unequal access to discounts and promotions based on predefined buying habits

Access to information and social circles, such as algorithms that promote news based on socio-economic groups and suggestions to join specific groups or circles

Employment discrimination, including algorithms that filter candidates based on their race, religion, or gender

Unequal use of police force and punishment, including algorithms that predict the possibility of an individual committing a crime in the future based on social status and race

Housing discrimination, including the denial of equal rental and mortgage opportunities to people of color, LGBT groups, and other minorities

AI has brought unprecedented benefits to our society similar to what the industrial revolution did. But with all these benefits, we should be aware of the societal changes that these benefits carry. If the future of driving is self-driving cars, this will mean that driving as a profession will disappear in the foreseeable future. Many other industries will be affected and will cease to exist. It does not mean that progress should not happen, but it needs to happen in a controlled way.

Software is only as perfect as its creators, and flaws in new AI-powered products are inevitable. But if these new applications are the first level in the decision-making process about human lives and destinies, there has to be a way to ensure that we minimize potential harmful consequences. Therefore, deeply understanding our models is paramount. Part of that is reproducibility, which is one of the key factors in minimizing the negative consequences of AI.

Summary

In this chapter, we have discussed a number of important concepts that help define why reproducibility is important and why it should be a part of a successful data science process.

We've learned that data science models are used to analyze historical data as input with a target goal to calculate the most probable and most successful result. We've established that replication, the ability to reproduce the results of a scientific experiment, is one of the fundamental principles of good research and that it is one of the best ways to ensure that your team is doing everything to reduce bias in your models. Bias can creep into a calculation from misrepresentation in a training dataset. Often, this reflects historical and social realities and norms accepted in society. Another way to reduce bias in your training data is to have a diverse team that includes representatives of all genders, races, and backgrounds.

We've learned that data dredging, or fishing, is an unethical technique used by some data scientists to prove a predefined hypothesis by cherry-picking the results of an experiment and only selecting the results that prove the desired outcome and ignoring any inconvenient trends.

We've also learned about the MLOps methodology, a lifecycle of a machine learning application, similar in its principle to the DevOps software lifecycle technique. MLOps includes the following main phases: planning, development, training, validation, deployment, and monitoring. All of the phases are continuously repeated, creating a feedback loop that ensures seamless experiment management from planning through development and testing to production and post-production phases.

We've also reviewed some of the most important aspects of ethical AI, a discipline of data science that focuses on ethical aspects of artificial intelligence, robotics, and data science. A failure to implement an ethical AI process in your organization might lead to undesirable legal consequences if deployed production models are found to be discriminatory.

In the next chapter, we will learn about the main concepts of the Pachyderm version-control system, which can help you address many of the issues described in this chapter.

Further reading

Raymond Perrault, Yoav Shoham, Erik Brynjolfsson, Jack Clark, John Etchemendy, Barbara Grosz, Terah Lyons, James Manyika, Saurabh Mishra, and Juan Carlos Niebles, The AI Index 2019 Annual Report, AI Index Steering Committee, Human-Centered AI Institute, Stanford University, Stanford, CA, December 2019: https://hai.stanford.edu/sites/default/files/ai_index_2019_report.pdf

Figure 1.1, 1.2, 1.3, 1.8, 1.9, and 1.11 were reproduced from the AI Index 2019 Annual Report.

Liu, D., & Salganik, M. (2019, March 15). Successes and struggles with computational reproducibility: Lessons from the Fragile Families Challenge. https://doi.org/10.1177/2378023119849803

Transparency and Openness Promotion (TOP) Guidelines v1.0.1: https://osf.io/ud578/

Reproducibility: A tragedy of errors: https://www.nature.com/news/reproducibility-a-tragedy-of-errors-1.19264

Neural Information Processing Systems (NeurIPS) conference: https://nips.cc

International Conference on Machine Learning (ICML): https://icml.cc

Chapter 2: Pachyderm Basics

Pachyderm is a data science platform that enables data scientists to create an end-to-end machine learning workflow that covers the most important stages of a machine learning life cycle, starting from data ingestion all the way into production.

If you are familiar with Git, a version control and life cycle system for code, you will find many similarities between the most important Git and Pachyderm concepts. Version control systems such as Git and its hosted version GitHub have become an industry standard for thousands of developers worldwide. Git enables you to keep a history of changes in your code and go back when needed. Data scientists deserve a platform that will let them track the versions of their experiments, reproduce results when needed, and investigate and correct bias that might crawl into one of the stages of the data science life cycle. Pachyderm provides benefits similar to Git that enable data scientists to reproduce their experiments and effortlessly manage the complete life cycle of the data science workflow.

This chapter is intended to describe the basics of Pachyderm architecture and its main concepts. We will cover the following topics:

Reviewing Pachyderm architecture

Learning about version control primitives

Discovering pipeline elements

Reviewing Pachyderm architecture

This section walks you through the distributed Pachyderm architecture and the internals of the Pachyderm solution. But before we dive into the nitty-gritty details of Pachyderm infrastructure, let's answer the question that a lot of you might have on your mind after reading the introduction—why can't I use Git or any other version control system? We'll address this question with Git in mind as it is the most popular and widely used software version control system, but all of the arguments apply to any other similar version control system for source code. After we review how Pachyderm is different and similar to Git, we will review the Pachyderm internals, Kubernetes, and container runtimes.

Why can't I use Git for my data pipelines?

So, if Pachyderm is similar to Git, why can't I store everything in Git rather than have multiple tools that I have to learn and support?, you might ask.

While Git is a great open source technology for software engineers, it is not tailored to solve the reproducibility problem in data science. This is mainly because Git was designed to track versions of files rather than establishing connections between data, code, and parameters – the essential parts of each data model.

While you can keep your code files in Git, doing so with the training data is not possible for the following reasons:

File format support: Git is great for source code, such as text files, scripts, and so on, but it does not version non-binary files, such as image or video files. Git will detect that a binary file has changed, but it does not provide detailed information about the changes like it does in the case of text files. In addition, Git replaces the whole binary file, which makes for a dramatic increase in your repository when you update it.

File size support: Often, image and video files are very large, and Git has a file size limitation of 100 MB per file. For best performance, Git recommends keeping your repository size below 1 GB, which might not be enough for some machine learning and deep learning projects.

Cloning a large Git repository and all the associated history of each file can be problematic and takes a long time.

Consider these limitations when you plan the tooling for your machine learning project.

In addition to standard Git repositories, Git offers storage for large files, called Git Large File Storage (LFS). However, this option likely won't provide the needed functionality either. Git LFS has the following disadvantages for machine learning projects:

Git LFS requires a server. You either need to set up and maintain your own Git LFS server or store it on a third-party online platform. Hosting your own server is problematic and requires additional support, and your preferred cloud provider might not be supported.

Online Git LFS servers have a limited free tier. For projects that use high-resolution images or videos, this likely means that the limit will be reached fairly soon, and an upgrade to a paid version will be required.

Designed for marketing department purposes, Git LFS seems like a good solution for marketing departments to store assets. Everyone else seems to prefer standard Git for their non-machine-learning projects.

Now that you know why source code version control systems are not the best solutions to work with data, let's review Pachyderm architecture.

Pachyderm architecture diagram

The following diagram shows the Pachyderm architecture:

Figure 2.1 – Pachyderm architecture

Let's review all the components of this diagram one by one.

Kubernetes

Kubernetes, an open source distributed workflow scheduling system, is at Pachyderm's core. If you are not familiar with Kubernetes, here is a quick overview.

Kubernetes is the latest stage of infrastructure evolution, which started with traditional hardware deployments where data center operators deployed all applications on one physical server. The applications would fight over resources, which resulted in poor performance in some applications. This didn't scale well and was difficult to manage. Later on, we saw the rise of virtual machine technology that made the management of physical servers much easier and allowed us to run multiple applications on one physical server with better scalability. Many data centers still run a lot of virtual machines. Containers, the technology on which Kubernetes is based, made the infrastructure even leaner by allowing containers to share common operating system components and removing the redundant copies of the same software parts. Containers can be easily ported from one cloud platform to another, provide better resource allocation, and overall are easier to manage and support.

The function of Kubernetes is to manage containers and collections of containers called Pods. Today, Kubernetes is the dominant container technology in the open source world. Big tech companies might have their own types of schedulers – for example, Google has a project called Borg, but the majority of others are using one or another version of Kubernetes.

Companies such as Amazon, Microsoft, and Google offer hosted versions of Kubernetes called, respectively, AWS Elastic Kubernetes Service (EKS), Microsoft Azure Kubernetes Service (AKS), and Google Kubernetes Engine (GKE). Kubernetes can also be installed on-premises by using an automated software provisioning tool, such as Ansible, or using one of the specifically designed Kubernetes installation tools, such as Kubespray.

Some companies, such as DigitalOcean, provide a Kubernetes installation and management system that can deploy Kubernetes to a cloud platform of your choice.

Now that we know what Kubernetes is, let's review how Pachyderm leverages Kubernetes to provide a scalable, reproducible data science solution.

Helm

The tool that deploys Pachyderm on top of Kubernetes is called Helm. Helm is an open source package manager that enables you to deploy any Kubernetes-compatible packages on a Kubernetes cluster. The main advantage of Helm is that it makes the deployment of packages easy and allows a lot of flexibility through Helm Charts. A Helm Chart is a YAML file where you specify parameters for your Pachyderm cluster. Pachyderm provides sample Helm Charts for all types of installations, including local and cloud deployments. You can find an example of the values.yaml that you can use with the Pachyderm Helm Chart for local Pachyderm installation at https://github.com/pachyderm/pachyderm/blob/master/doc/docs/master/reference/helm_values.md.

In the values.yaml file, you can define everything you want to deploy with your Pachyderm cluster. For example, you can deploy the Pachyderm UI called Console, or not; you can configure storage buckets in your cloud provider; you can specify which version of Pachyderm to install, and so on.

Helm Charts are stored in Artifact Hub, located at https://artifacthub.io/. You can find Pachyderm at https://artifacthub.io/packages/helm/pachyderm/pachyderm. We will discuss the Pachyderm Helm Chart in more detail in Chapter 5, Installing Pachyderm on a Cloud Platform.

Pachyderm internals

When you deploy Pachyderm locally or on a cloud platform of your choice, the underlying Kubernetes orchestrator deploys the following components:

Pachyderm Console (UI): The Pachyderm in-browser User Interface (UI) that provides basic Pachyderm functionality.

Pachd: The Pachyderm daemon container that is responsible for managing all the main Pachyderm logical operations.

postgress: Pachyderm requires a PostgreSQL instance to store metadata. PostgreSQL is a relational database popular in many open source and commercial solutions. While for testing purposes, an instance of PostgreSQL is shipped with the Pachyderm build, for production deployments and workloads, a separate cloud-based instance is strongly recommended.

etcd: In older versions of Pachyderm, etcd was the key-value store that stored information about nodes and administrative metadata. In versions of Pachyderm 2.0.0 and later, etcd only stores a small portion of metadata, while the majority is stored in PostgreSQL.

Each of these components is deployed as a Kubernetes Pod—the smallest deployment unit of Kubernetes, which can have one or a collection of containers. Unlike in other applications, in Pachyderm, each Pod has only one container.

In addition to these containers that are deployed during the installation, every Pachyderm pipeline, a computational component that trains your model, is deployed as a separate Kubernetes Pod. These Pods are called workers. You can deploy an unlimited number of pipelines in your Pachyderm cluster and assign a custom amount of resources to each Pod. Pods are completely isolated from each other.

Other components

Figure 2.1 has the following components that we have not yet listed:

Load Balancer or Ingress Controller: A networking component of Kubernetes that exposes HTTP/HTTPS routes from the cluster to the outside world. By default, Kubernetes provides a ClusterIP service that enables the services inside the cluster to talk to each other. There is no way to access cluster services from the outside world. In a production deployment, such as on a cloud platform, you need to deploy either an ingress controller or a load balancer to be able to access your Pachyderm cluster from the internet. In a local, testing deployment, Pachyderm uses port-forwarding. NodePort is another way to configure external access. However, it is not recommended for production deployments and, therefore, has been omitted from the current description. Pachyderm provides a Traefik ingress option to install through its Helm Chart.

Pachyderm CLI or pachctl: pachctl is Pachyderm's Command-Line Interface (CLI), which enables users to perform advanced operations with Pachyderm pipelines and configure infrastructure as needed.

API: Pachyderm supports programmatical access through a variety of language clients, including Python, Go, and others. Many users will find it useful to use one of the supported clients or even build their own.

Metadata storage: etcd collects administrative metadata, which has to be stored either on a local disk or a Kubernetes Persistent Volume (PV), which can be on a cloud platform or any other platform of your choice.

Object Storage: A location to store your Pachyderm File System (PFS) and the pipeline-related files. Pachyderm supports S3 object storage, such as MinIO, Google S3, Azure Blob storage, and Amazon S3.

In Figure 2.1, the user accesses the Pachyderm cluster through a load balancer or ingress controller by using either the Pachyderm dashboard, pachctl, or through the API by using a language client. The user's call is sent to pachd. pachd processes the request and updates the state of the pipeline and etcd accordingly. After the pipeline runs, it outputs the result to the configured storage location, from where it can be accessed by other third-party applications.

Container runtimes

A container runtime is software that executes the code in a container. There are many open source container runtimes, such as Docker, container, CRI-O, and others. Pachyderm supports the most popular container runtime—Docker. The fact that Pachyderm supports only one type of container runtime does not mean limited functionality. Most users will find Docker containers sufficient for all their needs.

To run a container, you need to have a container image and a place to store it. A container image is a file that contains the immutable code of a program that can be executed by the container runtime. Container images are stored in container registries, storage or repositories that host container images. Some container registries are public and others are private. All Pachyderm components are packaged as container images and stored in Docker Hub, a public container registry, meaning that anyone can download those images free of charge.

All your Pachyderm pipelines will have to be packaged as Docker containers and stored in a container registry. Your organization might require you to store your packages in a private container registry to protect the intellectual property of your pipeline. But if you are working on an open source or student project, you can store your container images in Docker Hub for free.

Now that we have reviewed the Pachyderm architecture, let's look at Pachyderm version control primitives that are foundational in understanding how Pachyderm works.

Learning about version control primitives

As we saw in a previous section, Pachyderm bears similarities to the code version control software called Git. If you have participated in developing an open source project before, you are likely familiar with Git through the use of a hosted Git version, such as GitHub, GitLab, or Gerrit.

Like with Git, you store your data in repositories, upload your data with a commit, and can have multiple branches in your repositories. Pachyderm stores the history of your commits and allows you to track changes or the history of your data back to its origins.

Pachyderm version control primitives enable you to go back in time and run your pipeline against previous versions of your changes. This can be very powerful in tracking bias and mistakes that crawl into your pipeline changes.

Let's look at these concepts in more detail.

Repository

A Pachyderm repository is a filesystem in which you store your data and where you store versions of your data. Pachyderm distinguishes between the input repository and output repository.

An input repository is a filesystem that you create and where you upload your data for further processing either by using a CLI, the UI, or automatically by using the API.

An output repository is a filesystem that Pachyderm creates automatically, which has the same name as the pipeline. This is the location where Pachyderm outputs the results of the calculation and from where results can be exported for serving as a model.

An important distinction from the Git repositories is that Pachyderm repository history is stored in a centralized location, which eliminates the risk of merge conflicts. Therefore, there is no equivalent of the .git history file in Pachyderm.

The following diagram demonstrates how internal and external repositories work within a pipeline:

Figure 2.2 – Pachyderm input and output repositories

The preceding diagram shows a simple use case with just one pipeline. However, your workflow could be much more complex with multiple pipelines being intertwined and working together to achieve the needed result.

Branch

A branch is a Pachyderm line of development that tracks a set of changes within a repository. By default, a repository has no branches. The master branch that we will use in this book is just an example of how you can name your primary branch in Pachyderm, and it's not enforced. Typically, you create a branch on a repository when you upload initial data in the repository. You can create multiple branches to organize your work, but you likely won't use them as extensively as in Git. Often, just one branch is enough for all the work. You could create a separate branch for a different experiment. All branches are visible to all users, so you won't be able to have a local branch, experiment in it, and then merge it with the master branch as you do in Git.

To create a branch, run the following:

pachctl create branch <repo>@<branch>

The repository must exist before you can create a branch.

To confirm that a branch was created, run the following:

pachctl list branch <repo>

The final part of this section will deal with the commit concept.

Commit

A commit is a single immutable changeset of your data. For example, when new data is received from a streaming platform, such as Kafka, it is written to the Pachyderm repository as a new commit with an individual hash identifier.

The following diagram illustrates the concept of the commit on a branch in Pachyderm:

Figure 2.3 – A diagram showing Pachyderm commits

Data changes are stored in the repository history. The HEAD, or the HEAD commit of the branch, moves every time a new commit is submitted to the repository. As soon as the new data is submitted, the pipeline runs the code against these latest changes unless a different behavior is explicitly configured.

Now that we know the most important Pachyderm version control primitives, let's learn about the Pachyderm pipeline in more detail.

Discovering pipeline elements

This section walks you through the main Pachyderm pipeline concepts. The Pachyderm Pipeline System (PPS) is the centerpiece of Pachyderm functionality.

A Pachyderm pipeline is a sequence of computational tasks that data undergoes before it outputs the final result. For example, it could be a series of image processing tasks, such as labeling each image or applying a photo filter. Or it could be a comparison between two datasets or a finding similarities task.

A pipeline performs the following three steps:

Downloads the data from a specified location.

Applies the transformation steps specified by your cod.

Outputs the result to a specified location.

The following diagram shows how a Pachyderm pipeline works:

Figure 2.4 – Pachyderm pipeline

Each Pachyderm pipeline has an input and output repository. An input repository is a filesystem within Pachyderm where it is being placed from an outside source. This is the location inside of the pipeline Pod under the /pfs directory. The data can either be pulled by a pipeline or pushed to the input repository by the data source system. After the data goes through the transformation, it is placed into the output Pachyderm repository, which is located in the /pfs/out directory. From the output repository, the results can be further consumed by third-party applications or other pipelines.

Every time new data lands in the input repository, Pachyderm starts a pipeline job. The job processes the newly arrived data and places the results in the output repository.

Types of pipelines

A machine learning pipeline continuously performs the tasks that the data scientists coded on the new data. For example, you might want to compare or join two types of files together or apply certain parameters to them. To simplify these tasks, Pachyderm offers predefined pipelines that can do those things automatically.

Pachyderm offers the following types of pipelines:

Standard pipeline

Cron pipeline

Spout pipeline

Service pipeline

To create a pipeline, you need to write a pipeline specification. A pipeline specification is a file in the YAML Ain't Markup Language (YAML) or JavaScript Object Notation (JSON) format that describes what the pipeline needs to do. We will talk about pipeline specifications in more detail in the next chapter.

Standard pipelines

A standard pipeline, or simply a pipeline, is the most straightforward way to schedule work in a Pachyderm cluster. This type of pipeline is triggered when new data lands in the Pachyderm input repository. The pipeline spawns or resumes a Kubernetes Pod to run your code against the newly landed data and outputs the result to the output repository. The output repository is created automatically and has the same name as the pipeline.

The simplest standard pipeline must have the following components:

name: A descriptive name for your pipeline. It is good practice to give your pipeline a name that reflects the task it accomplishes. For example, if your code analyzes the sentiment of users' feeds on social media, you might want to call it sentiment-analysis or something similar.

transform: The transformation section of a pipeline contains the information about the Docker image that the pipeline needs to pull and use and the code that it needs to run in the pipeline container. For example, if you are doing sentiment analysis, you probably will use Natural Language Processing (NLP) tools such as Natural Language Toolkit (NLTK) or Stanford CoreNLP. Therefore, you can use either an already available Docker image for those tools or build your own custom one.

input: An input repository from which the pipeline grabs data for processing. You will need to upload the data to an input repository either through the CLI, UI, or API.

The following text is an example of a pipeline specification that performs sentiment analysis. The pipeline is called sentiment-analyzer, uses the Docker image called ntlk-image, downloads data from the input repository called feed, and then runs the code stored in the file called transform_code.py:

# sentiment-analyzer.yml

---

pipeline:

  name: sentiment-analyzer

transform:

  image: nltk-image

  cmd:

  - python3

  - "/transform_code.py"

input:

  pfs:

    repo: feed

    glob: "/"

The glob parameter defines how to break down data into processable chunks that can be spread across multiple Pachyderm workers for better performance. We will discuss this parameter in more detail in the next section.

Important note

You can either put your transformation code in a file like in the example above or specify it directly in the cmd field. See Chapter 3, Pachyderm Pipeline Specification, for more information.

Cron pipelines

A Cron pipeline, or cron, is a pipeline that runs a specific task periodically. If you are familiar with the UNIX software utility cron, the Cron pipeline uses a similar logic—it runs periodically according to the specified schedule and performs the same task every time. Unlike the standard pipeline that runs every time new data arrives in the input repository, the Cron pipeline runs according to a schedule. You might want to use this pipeline to periodically scrape a website or a table.

A Cron pipeline specification would look very similar to the standard pipeline apart from the input section. You need to specify the time interval in the input section for your pipeline.

The following text is an example of a Cron pipeline in YAML format that scrapes a website called my-website once a day at midnight:

# scrape-my-website.yml

---

pipeline:

  name: scrape-my-website

transform:

  image: web-scraper

  cmd:

  - python3

  - "/scraper.py"

input:

  cron:

    repo: mywebsite

    spec: "@daily"

All the information about the website URL will go into your scraper.py file.

Spout pipelines

A spout pipeline, or spout, as the name suggests, is designed to stream data from an outside source, such as a publish/subscribe messaging system. In your infrastructure, you might have multiple message queue systems. With a Pachyderm spout, you can consolidate inputs from all of these systems and inject them into a Pachyderm spout pipeline. The spout code is not triggered by any event; instead, it runs continuously, listening for new messages.

Unlike in a standard or a Cron pipeline, spouts do not have an input repository, but instead, they listen on a specified address. The port and host can be specified either as an env variable in the pipeline specification or inside of the container.

Often, a spout is used together with a Pachyderm Service pipeline to expose the results of the pipeline because the data in the spout's output repository cannot be accessed with standard Pachyderm commands.

The following text is an example of a spout pipeline in YAML format that connects to an Amazon Simple Queue Service (SQS) host and applies a Python script, myscript.py, to the messages:

---

pipeline:

  name: spout-pipeline

spout: {}

transform:

  cmd:

  - python3

  - myscript.py

  image: 'myimage'

  env:

    HOST: sqs-host

    TOPIC: mytopic

    PORT: '1111'

The spout section is left empty, but this is the place where you can combine the spout with a service pipeline to expose the results of the pipeline to the outside world.

Service pipelines

A service pipeline, or service, is a pipeline that can expose your output repository to the outside world. Unlike other pipelines, it does not perform any modifications to your data. The only function of this pipeline is to serve the results for your pipeline as an API, for example, in the form of a dashboard. A service pipeline is often combined with a spout pipeline and is sometimes called spout-service. However, it can use the results from any other pipeline output repository as its own input repository to expose the results.

A pipeline specification for this pipeline misses the transformation section. The following text is an example of a service pipeline in YAML format:

---

pipeline:

  name: expose-service

input:

  pfs:

    glob: "/"

    repo: final-results

service:

  external_port: 31111

  internal_port: 8888

Next, we'll see how Pachyderm works with datum.

Datum

A datum is the smallest piece of data that a Pachyderm worker can process. Folders and files in your input repository can be processed all as one datum or as multiple datums. This behavior is controlled by the glob parameter in the pipeline specification. Pachyderm processes each datum independently. If you have more than one worker running your pipeline, Pachyderm can schedule datums to run on separate workers for faster processing and, in the end, all datums are merged back together in the output repository.

Datums

This is the term used by the Pachyderm folks. Yes, usually, the plural is data, but datums is how it appears throughout Pachyderm's documentation!

Breaking down your data into multiple datums provides the following advantages:

Improves performance by scaling your pipeline to multiple Pachyderm workers

Enables you to process only specific files and folders

For example, say you have a repository with the following folder structure:

data

├── folder1

│   ├── file1

│   ├── file2

│   └── file3

├── folder2

│   ├── file1

│   └── subfolder1

│       └── file1

└── folder3

    ├── subfolder1

    │   ├── file1

    │   └── file2

    └── subfolder2

        ├── file1

        └── file2

For this folder structure, you can set the following types of glob patterns:

/: Process all files and folders as a single datum. This single datum includes all files and folders in the / directory.

/*: Process each folder as a separate datum. The resulting number of datums is three, including the following:

/folder1

/folder2

/folder3

/*/*: Process each filesystem object on the /*/* level as a separate datum. The resulting number of datums is seven, including the following:

/folder1/file1

/folder1/file2

/folder1/file3

/folder2/file1

/folder2/subfolder1

/folder3/subfolder1

/folder3/subfolder2

/*/*/*: Process each filesystem on the third level. The resulting number of datums is five, including the following:

/folder2/subfolder1/file1

/folder3/subfolder1/file1

/folder3/subfolder1/file2

/folder3/subfolder2/file1

/folder3/subfolder2/file2

/**: Process each file, folder, and subfolder as a separate datum. The resulting number is 15, which includes all files and folders listed in the preceding code extract:

/folder1/

/folder1/file1

/folder1/file2

/folder1/file3

/folder2/

/folder2/file1/

/folder2/subfolder1/

...

/<filesystem>/*: Process only the files and folders that match the naming pattern. For example, if you decide to process only the data in folder1, you will set your glob pattern to /folder1/*. Similarly, you can set just the first few letters of a directory name as a glob pattern.

These are the most commonly used glob patterns. Extended glob patterns are available with the Pachyderm's ohmyglob library. For more information, see https://github.com/pachyderm/ohmyglob.

Scaling your pipeline with datums

One of the biggest advantages of breaking your data into multiple datums is the ability to scale your work across multiple pipeline workers to significantly improve the performance and the processing time of your pipeline. By default, Pachyderm deploys one worker node for each pipeline, but you can increase this number as needed by specifying the parallelism_spec parameter in the pipeline specification.

The following diagram demonstrates how datums are scaled across multiple workers:

Figure 2.5 – Scaling a pipeline

In the preceding diagram, the input data is broken down into three workers, which simultaneously start the processing. After all of the datums are processed, they are merged into the final output result. It is important to emphasize that datums only exist inside of a Pachyderm pipeline. They cannot be accessed, mounted, or modified in any way.

Pachyderm inputs

In Pachyderm, you likely will create multiple repositories to store training data, testing data, and parameters. Pachyderm provides an automated way to combine files in different repositories in the pipeline to be processed together through the use of inputs. If you have used SQL before, you might find that these types of inputs remind you of SQL operators. However, there is one major difference that distinguishes SQL operators from Pachyderm inputs. While in SQL you can create matching pairs of rows in each table, Pachyderm inputs work on the file level only. This means that you can only combine files or directories inside Pachyderm repositories based on their names rather than the contents of the files.

Important note

This section provides examples of how you can create various Pachyderm inputs. These examples are for your reference only and you might want to use some of them in your future Pachyderm work after you install and configure Pachyderm. We will cover the installation and setup in the Installing Pachyderm locally and Deploying Pachyderm on a cloud platform sections of Chapter 3, Pachyderm Pipeline Specification.

Cross inputs

A cross input, or cross, is an input in which each file from one repository is combined with a file in the other repository. The set of matched files is determined by the glob pattern, and all that data is visible to the pipeline code at the time of the pipeline run.

The input part of your pipeline file in YAML format might look like this:

…

input:

  cross:

  - pfs:

      glob: "/*"

      repo: data

  - pfs:

      glob: "/"

      repo: parameters

…

For example, you have two Pachyderm repositories, data and parameters, with the following structure and files:

data

├── image1.png

├── image2.png

├── image3.png

└── image4.png

parameters

├── param1.csv

└── param2.csv

If you create a pipeline that creates a cross-product of these repositories with a glob pattern set to /*, you will get eight datums in total—each file from the data repository will be combined with each file in the parameters repository. This is how the files will be processed:

data@4e5897...:/image1.png, parameters@d1a0da...:/param1.csv

data@4e5897...:/image2.png, parameters@d1a0da...:/param1.csv      

data@4e5897...:/image3.png, parameters@d1a0da...:/param1.csv      

data@4e5897...:/image4.png, parameters@d1a0da...:/param1.csv       

data@4e5897...:/image1.png, parameters@d1a0da...:/param2.csv       

data@4e5897...:/image2.png, parameters@d1a0da...:/param2.csv      

data@4e5897...:/image3.png, parameters@d1a0da...:/param2.csv      

data@4e5897...:/image4.png, parameters@d1a0da...:/param2.csv      

In the preceding output, each pair of files and each line is one datum.

Union inputs

A union input, or union, is an input that enables you to combine datums from one repository with datums in another repository. The total number of datums is the sum of datums in all repositories. If we take the same repository structure as described in the Cross inputs section and, instead of creating a cross-product, just add them to each other by using the union input, we will have a total of five datums.

The input part of your pipeline file in YAML format might look like this:

…

input:

  union:

  - pfs:

      glob: "/*"

      repo: data

  - pfs:

      glob: "/"

      repo: parameters

…

The following output demonstrates the list of datums for a pipeline with the /* glob pattern set for both repositories:

data@4e58978ca1304f16a0a5dfe3715363b4:/image1.png      

data@4e58978ca1304f16a0a5dfe3715363b4:/image2.png

data@4e58978ca1304f16a0a5dfe3715363b4:/image3.png

data@4e58978ca1304f16a0a5dfe3715363b4:/image4.png

model@1de402b7be004a289f6d7185b2329b21:/

Your code processes each file individually and recognizes only one file with every run, ignoring all other files.

Join inputs

A join input, or join, is a Pachyderm input that enables you to match pairs of files based on a specific naming pattern defined by a capturing group. A capturing group is a regular expression that is defined within parentheses. The capturing group must be specified in the join_on parameter in your Pachyderm pipeline.

Pachyderm separates inner joins and outer joins. The difference between the two is that an inner join matches pairs of files, skipping files that do not match the specified pattern. An outer join acts the same way as an inner join, but also includes files that do not match the pattern in the pipeline run. If this sounds a bit confusing, the following example should clarify how the pipeline works.

Let's say you have two repositories, data and parameters, and the parameters repo has the following structure:

NAME                 TYPE SIZE

/param-0101-2021.txt file 4B

/param-0102-2021.txt file 4B

/param-0103-2021.txt file 4B

/param-0104-2021.txt file 4B

/param-0105-2021.txt file 4B

The data repository has the following structure:

NAME                TYPE SIZE

/data-0101-2021.txt file 2B

/data-0102-2021.txt file 2B

/data-0103-2021.txt file 2B

/data-0104-2021.txt file 2B

/data-0105-2021.txt file 2B

/data-0106-2021.txt file 2B

/data-0107-2021.txt file 2B

In your pipeline, you might want to match the files by their date. To do so, you would have to specify a capturing group of $1 and a glob pattern, /. Here is an example of a pipeline specification in YAML format that matches these file paths:

---

pipeline:

   name: describe

input:

   join:

   - pfs:

       glob: "/data-(*).txt"

       repo: data

       join_on: "$1"

   - pfs:

       glob: "/param-(*).txt"

       repo: parameters

       join_on: "$1"

transform:

   image: mydockerhub/describe

   cmd:

   - python3

   - "/describe.py"

This capturing group in combination with the / glob pattern matches five pairs of files:

data@2c95b1...:/data-0101-2021.txt, parameters@b7acec...:/param-0101-2021.txt

data@2c95b1...:/data-0102-2021.txt, parameters@b7acec...:/param-0102-2021.txt

data@2c95b1...:/data-0103-2021.txt, parameters@b7acec...:/param-0103-2021.txt

data@2c95b1...:/data-0104-2021.txt, parameters@b7acec...:/param-0104-2021.txt

data@2c95b1...:/data-0105-2021.txt, parameters@b7acec...:/param-0105-2021.txt

/data-0106-2021.txt and /data-0107-2021.txt do not have matching pairs, and therefore, Pachyderm would skip them in this run. However, if you want to include these in the files in the pipeline run, you can specify outer_join: true in the data repository input to include these files in the pipeline run without a pair. This abstract shows how you can add this parameter:

…

input:

   join:

   - pfs:

       glob: "/data-(*).txt"

       repo: data

       join_on: "$1"

       outer_join: true

…

Then the list of datums in your pipeline will look like this:

data@2c95b1...:/data-0101-2021.txt, parameters@b7acec...:/param-0101-2021.txt

data@2c95b1...:/data-0102-2021.txt, parameters@b7acec...:/param-0102-2021.txt

data@2c95b1...:/data-0103-2021.txt, parameters@b7acec...:/param-0103-2021.txt

data@2c95b1...:/data-0104-2021.txt, parameters@b7acec...:/param-0104-2021.txt

data@2c95b1...:/data-0105-2021.txt, parameters@b7acec...:/param-0105-2021.txt

data@2c95b1...:/data-0106-2021.txt

data@2c95b1...:/data-0107-2021.txt

data-0106-2021.txt and data-0107-2021 are included in the pipeline run without a pair.

Group inputs

A group input, or group, is a Pachyderm input that enables you to group matching files within one or more repositories based on a specific naming pattern. This type of input is similar to joins but it can address a broader use case. In a group input, you must specify a replacement group under the group_by pipeline parameter. If we took the same pipeline that we used in the inner join input example and replaced join with group and join_on with group_by, we would get the exact same result as we got with the inner join input.

The main differences from the join input include the following:

A group input creates one large datum of the files that match a naming pattern, while a join crosses two files that match a pattern and typically creates more datums.

A group input enables you to match pairs in a single repository, while a join requires at least two repositories.

A group input is primarily useful in time series scenarios when you need to group files based on their timestamps.

Therefore, you could create a cron pipeline with the replacement group that matches a specific timestamp.

For example, say you have repository data with the following structure:

/data-0101-2020.txt file 2B

/data-0101-2021.txt file 2B

/data-0102-2020.txt file 2B

/data-0102-2021.txt file 2B

/data-0103-2020.txt file 2B

/data-0103-2021.txt file 2B

/data-0104-2021.txt file 2B

/data-0105-2021.txt file 2B

/data-0106-2021.txt file 2B

/data-0107-2021.txt file 2B

Your pipeline in YAML format might look like this:

---

pipeline:

   name: test-group

transform:

   cmd:

   - python3

   - "/test.py"

input:

   group:

   - pfs:

       repo: data

       glob: "/data-(*)-(*).txt"

       group_by: "$1"

You have two replacement groups listed in the pipeline. Replacement group one matches the month and day in the name of the text file. For example, in the file /data-0104-2021.txt, it is 0104. The second replacement group matches the year in the timestamp. In the same file, it is 2021.

If you specify the first matching group in your pipeline, the resulting list of datums will include three pairs and seven datums in total:

data@...:/data-0101-2020.txt, data@..:/data-0101-2021.txt

data@...:/data-0102-2020.txt, data@...:/data-0102-2021.txt

data@...:/data-0103-2020.txt, data@...:/data-0103-2021.txt

data@...:/data-0104-2021.txt

data@...:/data-0105-2021.txt                                                         

data@...:/data-0106-2021.txt                                                            

data@...:/data-0107-2021.txt

In the preceding output, the files with the matching day and month, such as 0101, are grouped in one datum. If you change the group_by parameter to use the second replacement group, meaning grouping by year, the list of datums will include two datums, grouping the files by year:

-  data@ecbf241489bf452dbb4531f59d0948ea:/data-0101-2020.txt, data@ecbf241489bf452dbb4531f59d0948ea:/data-0102-2020.txt, data@ecbf241489bf452dbb4531f59d0948ea:/data-0103-2020.txt                                                                                                                                                                                                                                             

-  data@ecbf241489bf452dbb4531f59d0948ea:/data-0101-2021.txt, data@ecbf241489bf452dbb4531f59d0948ea:/data-0102-2021.txt, data@ecbf241489bf452dbb4531f59d0948ea:/data-0103-2021.txt, data@ecbf241489bf452dbb4531f59d0948ea:/data-0104-2021.txt, data@ecbf241489bf452dbb4531f59d0948ea:/data-0105-2021.txt, data@ecbf241489bf452dbb4531f59d0948ea:/data-0106-2021.txt, data@ecbf241489bf452dbb4531f59d0948ea:/data-0107-2021.txt

In this section, we have learned about the most important Pachyderm pipeline primitives and how they differ from each other.

Summary

In this chapter, we have learned about the most important Pachyderm version control primitives, including repositories, branches, and commits. We reviewed that although they are similar to those of the Git version control system, they have a few very important differences.

We've learned about the types of Pachyderm pipelines and inputs and how a pipeline can be scaled and optimized through the use of glob patterns and datums.

In the next chapter, we will review the Pachyderm pipeline specification in more detail and will learn how to use the various pipeline settings to run a pipeline in the most efficient way.

Further reading

Git documentation: https://git-scm.com/

Kubernetes documentation: https://kubernetes.io/docs/home

Helm documentation: https://helm.sh/docs/

Chapter 3: Pachyderm Pipeline Specification

A Machine Learning (ML) pipeline is an automated workflow that enables you to execute the same code continuously against different combinations of data and parameters. A pipeline ensures that every cycle is automated and goes through the same sequence of steps. Like in many other technologies, in Pachyderm, an ML pipeline is defined by a single configuration file called the pipeline specification, or the pipeline spec.

The Pachyderm pipeline specification is the most important configuration in Pachyderm as it defines what your pipeline does, how often it runs, how the work is spread across Pachyderm workers, and where to output the result.

This chapter is intended as a pipeline specification reference and will walk you through all the parameters you can specify for your pipeline. To do this, we will cover the following topics:

Pipeline specification overview

Understanding inputs

Exploring informational parameters

Exploring transformation

Optimizing your pipeline

Exploring service parameters

Exploring output parameters

Pipeline specification overview

Typically, when you conduct an ML experiment, it involves multiple sequential steps. In the simplest scenario, your pipeline takes input from an input repository, applies your transformation code, and outputs the result in the output repository. For example, you may have a set of images to apply a monochrome filter to and then output the result in an output repository that goes by the same name as the pipeline. This workflow performs only one operation and can be called a one-step pipeline, or one-step workflow. A diagram for such a pipeline would look like this:

Figure 3.1 – One-step workflow

The specification for this simple pipeline, in YAML format, would look like this:

---

pipeline:

  name: apply-photo-filter

transform:

  cmd:

  - python3

  - "/photo-filter.py"

  image: myregistry/filter

input:

  pfs:

    repo: photos

    glob: "/"

This is the simplest pipeline specification that you can create. It must include the following parameters:

name: A descriptive name for your pipeline. Often, the name of the pipeline describes a step in your ML workflow. For example, if this pipeline applies a photo filter to your images, you can call it apply-photo-filter. Alternatively, if it validates your model, you could call it cross-validation.

transform: This parameter includes your transformation code, which can be specified as a reference to a file or directly inline. We will discuss this parameter in more detail in the next section.

input: This parameter refers to an existing input repository where the files for the pipeline are located. input is a filesystem inside your pipeline worker under the pfs/ directory. For example, if your repository is called photos, it is stored under pfs/photos on your pipeline worker. Output repositories are created automatically by the pipeline and stored under pfs/out. All output repositories have the same name as the pipeline. For example, if your pipeline is called apply-photo-filter, the output repository will be stored as apply-photo-filter in pfs/out/.

This is a very simple example of a pipeline. In a more realistic use case, you would likely have more than one pipeline step. In a typical ML pipeline, you need to perform pre-processing, training, and cross-validation steps, among others. When you have multiple pipelines chained together, this is called a multi-step workflow. For example, if you are creating an NLP pipeline, your pipeline may have the following structure:

Figure 3.2 – Multi-step workflow

In the preceding diagram, each pipeline has a pipeline specification with a name, input repository, transformation code, and other parameters defined.

All pipeline specifications must be written in YAML Ain't Markup Language (YAML) or JavaScript Object Notation (JSON) format. These formats are easy for people to read and write and are widely used in various configuration files in the industry. It is easier to write in than Extensible Markup Language (XML) or other similar formats.

Now that we have reviewed the minimum pipeline specification and have looked at a more realistic example, let's review the other parameters that you can specify.

Understanding inputs

We described inputs in Chapter 2, Pachyderm Basics, in detail by providing examples. Therefore, in this section, we'll just mention that inputs define the type of your pipeline. You can specify the following types of Pachyderm inputs:

PFS is a generic parameter that defines a standard pipeline and inputs in all multi-input pipelines.

Cross is an input that creates a cross-product of the datums from two input repositories. The resulting output will include all possible combinations of all datums from the input repositories.

Union is an input that adds datums from one repository to the datums in another repository.

Join is an input that matches datums with a specific naming pattern.

Spout is an input that consumes data from a third-party source and adds it to the Pachyderm filesystem for further processing.

Group is an input that combines datums from multiple repositories based on a configured naming pattern.

Cron is a pipeline that runs according to a specified time interval.

Git is an input that enables you to ingest data from a Git repository.

For all inputs except for Cron and Git, you can define a pfs parameter that defines the input.

pfs

The pfs parameter, which stands for Pachyderm File System, defines the input and all its attributes, such as name, repository, branch, and others. A simple Pachyderm pipeline takes only one input, while the multi-input pipelines take multiple pfs inputs. You need to define one or multiple pfs inputs for all pipelines, except for Cron and Git.

Here are the sub-parameters of the pfs input:

name: The name of your pipeline.

repo: A Pachyderm input repository where the data is stored.

branch: A branch in the Pachyderm input repository where the data is stored. Often, this will be the master branch.

glob: A parameter that defines how to break the data into chunks for processing. You can read more about the glob parameter in the Datums section of Chapter 2, Pachyderm Basics.

lazy: A parameter that enables slower, less aggressive data downloading on a pipeline. The lazy parameter is useful when you need to look into a subset of your data.

s3: This parameter defines whether to include an S3 gateway sidecar on the pipeline. When you integrate with third-party applications through the S3 gateway, this ensures that the pipeline's provenance is preserved.

You can read more about the types of pipelines and inputs and view example pipelines in Chapter 2, Pachyderm Basics. The next section describes informational parameters that you can define for your pipeline.

Exploring informational parameters

Pipeline informational parameters define basic information about the pipeline. Out of all of them, only the name parameter is required in any pipeline specification. All other parameters are optional and can be omitted. Let's look at these parameters in more detail.

name

The name parameter is the descriptive name of your pipeline. Typically, you want to name a pipeline after the type of transformation it performs. For example, if your pipeline performs image classification, you may want to call it image-classification. A pipeline name must consist of alphanumeric characters, dashes, and underscores, and cannot exceed 63 symbols.

The following is an example of the name parameter in YAML format:

---

pipeline:

  name: image-classification

The following is an example of the name parameter in JSON format:

{

  "pipeline": {

    "name": "image-classification"

  },

Next, let's look at the description parameter.

description

The description parameter provides additional information about the pipeline. Although it is an optional parameter, it is good practice to add a short description to your pipeline. For example, if your pipeline performs image classification, you can add the following description: A pipeline that performs image classification by using scikit-learn.

The following is an example of the description parameter in YAML format:

description: A pipeline that performs image classification by using scikit-learn.

The following is an example of the description parameter in JSON format:

"description": "A pipeline that performs image classification by using scikit-learn.",

Next, let's learn about metadata.

metadata

The metadata parameter enables you to specify Kubernetes labels or annotations. Labels are typically used to group Kubernetes objects into some sort of category and help simplify the management of those objects. Labels can be queried to display objects of the same type.

Annotations, on the other hand, can be used to specify any random key-value pairs that are not defined within Kubernetes and could be used by external applications. You can use annotations to define the type of service, but things such as Identity Access Management (IAM) roles should be specified through pach_patch or pach_spec instead. Multiple labels and annotations can be specified in each pipeline specification.

Here is an example of how to specify annotations in YAML format:

metadata:

  annotations:

    annotation: data

The following example shows how to specify annotations in JSON format:

"metadata": {

    "annotations": {

        "annotation": "data"

    }

  },

The following example shows how to specify labels in YAML format:

metadata:

  labels:

    label: object

Finally, the following example shows how to specify labels in JSON format:

"metadata": {

     "labels": {

        "label": "object"

    }

  },

Now that we've learned about the informational parameters, let's look at the transformation section of the Pachyderm pipeline.

Exploring transformation

The transformation section is where you define your pipeline transformation code. It is the core of your pipeline's functionality. Most pipelines, unless they are a connector between two pipelines or a pipeline that exports results outside of Pachyderm, must have a transformation section.

The most important parameters of a transformation section – and the ones that are most commonly used – are image and cmd or stdin, env, and secrets.

Let's look at these parameters in more detail.

image

The image parameter defines a Docker image that your pipeline will run. A Docker image contains information about the environment in your pipeline container. For example, if you are running Python code, you will need to have some version of Python in your pipeline image. There are many publicly available containers that you can use for your pipeline.

You can also include your scripts in that container. Unless your code is just a Bash script that can be specified through the stdin parameter inline, you will likely need to build your own Docker image, include your code in that image, and store it in a public or private container registry. Docker images are built from a Dockerfile, which describes the container environment and what you can run in the container. You can read more about Docker images at https://docs.docker.com/.

Important note

Do not use the Docker CMD instruction; instead, use RUN. The CMD instruction will fail.

The following code shows how to specify a Docker image in YAML format:

metadata:

  labels:

    label: object

The following code shows how to specify a Docker image in JSON format:

"transform": {

    "image": "my-image",

However, just specifying a Docker image is not enough. You must define what to run, either through the cmd or stdin parameter.

cmd

The cmd parameter defines the code that the pipeline will run against your data. There is a lot of flexibility around what you can put in the cmd line. Typically, you want to specify the type of command you want to run, such as python, or set it to run a command-line shell, such as Bourne Shell (sh) or Bourne Again Shell (bash), and then specify the list of commands that you want to run in the stdin parameter.

There is no difference or preference between the two approaches. The only difference is that if you specify a file in the cmd parameter, you will need to build a Docker image and include that file in the image.

For example, if you have a Python 3 file that contains the code that you want to run against your data, you can specify it like this in YAML format:

cmd:

  - python3

  - "/test.py"

Alternatively, you can specify labels in JSON format:

"transform": {

    "cmd": [ "python3", "/test.py" ],

However, if you want to specify your commands inline in the stdin parameter, just have the format specified in the cmd parameter like this, in YAML format:

cmd:

  - python3

You can do the same in JSON format:

"transform": {

    "cmd": [ "python3" ],

See the stdin section for examples of how you can specify your inline code.

Your cmd field can get even more complex than that, however. For example, you can specify a script inside the cmd parameter.

The following text is an example of the syntax you can use in the cmd field in YAML format:

transform:

  image: my-image

  cmd:

  - tree

  - "/pfs"

  - "&&"

  - python

  - my-script.py

  - "--outdir"

  - "/pfs/out"

  - "--input"

  - "/pfs/data "

The following text is an example of the syntax you can use in the cmd field in JSON format:

"transform": {

    "image": "my-image",

    "cmd": ["tree",

        "/pfs",

        "&&",

        "python",

        "my-script.py",

        "--outdir", "/pfs/out",

              "--input", "/pfs/data "]

  },

Next, let's review the stdin parameter.

stdin

The stdin parameter is similar to the UNIX standard input (stdin), and it enables communication between the Pachyderm environment and the pipeline worker. This means that you can put a code in the format specified in the cmd command inline in the stdin field. You could also specify paths to your code files as well, similar to the cmd parameter.

This method does not require you to build a Docker image and allows you to configure your pipeline entirely through the pipeline specification. If you are unfamiliar with the Docker image-building process, this approach may feel more appealing. However, for more complex pipelines, you likely want to save your scripts in files, build Docker images, and use them in your pipeline.

The following code shows the syntax you can use in the stdin field in YAML format:

transform:

  cmd:

  - bash

  stdin:

  - for f in /pfs/data/*

  - do

  - filename=$(basename "$f")

  - cp /pfs/data/* pfs/out/mypipeline/*

  - done

The following is an example of the syntax you can use in the stdin field in JSON format:

"transform": {

    "cmd": ["bash"],

    "stdin": [

        "for f in /pfs/data/*",

        "do",

        "filename=$(basename \"$f\")",

        "cp /pfs/data/* pfs/out/mypipeline/*",

        "done"]

  },

Because the preceding examples do not reference any files, you do not need to build a specific Docker image for them and include the file in there.

However, if you do reference any files or any environment prerequisites that are even slightly more complex than Bash, you likely need a Docker image. For example, if you have a my-script.py file that contains your code, you need to build a Docker image that will include that script, and you must reference it in your pipeline specification.

err_cmd

The err_cmd parameter enables you to define how Pachyderm handles failed datums and ultimately allows you to treat failed datums as non-critical errors, allowing a job with failed datums to succeed and trigger the next job only with the healthy datums. err_cmd does not write any data to the output repo. The err_cmd field is often used in combination with the err_stdin field, where you specify the actual error code, though you could also refer to a file with your error code. If you want your pipeline to succeed even when the job contains failed datums, you can simply set "err_cmd": true.

The following is the syntax you can use in the err_cmd field in YAML format:

transform:

  ...

  err_cmd:

  - bash

  - "/my-error-code.sh"

The following is the syntax you can use in the err_cmd field in JSON format:

"transform": {

    ...

    "err_cmd": [ "bash", "/my-error-code.sh"],

We will look at an example of how to use err_cmd in combination with err_stdin in the next section.

err_stdin

The err_stdin parameter is used in combination with the err_cmd parameter to specify the error code to run against failed datums. Similar to the stdin parameter, you can specify inline code to handle failed datums. For example, you may want to check if a datum is in a specific directory, and if it is, mark the datum as failed. Typically, you can just write a simple Bash script with an if… then condition to handle this.

The following code shows the syntax you can use in err_stdin with the err_cmd field in YAML format:

transform:    

    ...

    err_cmd:

    - bash

    err_stdin:

    - if

    - "[-a /pfs/repo1]"

    - then

    - exit 0

    - fi

    - exit 1

The following code shows the syntax you can use in err_stdin with the err_cmd field in JSON format:

"transform": {

    ...

    "err_cmd": [

      "bash"

    ],

    "err_stdin": [

      "if",

      "[-a /pfs/repo1]",

      "then",

      "exit 0",

      "fi",

      "exit 1"

    ]

Next, let's learn about the env parameter.

env

The env parameter enables you to specify Pachyderm the environment variables and arbitrary variables that you need to communicate with other third-party tools. These parameters may include paths to directories and files, hostnames and ports, secret access keys, various identifiers, and many others.

Pachyderm variables can be included as well. For example, you can use the LOG_LEVEL environment variable to specify the verbosity of your log messages for pachd. As another example, you can also specify an AWS region and a bucket in the env field.

The following code shows the syntax you can use in the env field in YAML format:

transform:    

    ...

    env:

        AWS_REGION: us-west-2

        S3_BUCKET: s3://my-bucket/

The following code shows the syntax you can use in the env field in JSON format:

"transform": {

    ...

    "env": {

             "AWS_REGION": "us-west-2",

             "S3_BUCKET": "s3://my-bucket/"

         }

  },

For a complete list of Pachyderm variables, see the Pachyderm documentation at https://docs.pachyderm.com/latest/deploy-manage/deploy/environment-variables/.

secrets

The secrets field enables you to specify Kubernetes secrets, which include sensitive information. This can include passwords or SSH public keys. You need to define a secret by either using the env_var and key parameters or the mount_point parameter.

The following code shows the syntax you can use in the name and mount_path fields to set the path to the secrets file in YAML format:

transform:    

    ...

    secrets:

        name: my-ssh-key

        mount_path: "/path/to/file"

The following code shows how to specify these parameters in JSON format:

transform:    

    ...

    "secrets": {

        "name": "my-ssh-key",

        "mount_path": "/path/to/file"

    }

The following code shows the syntax you can use in the env_var and key parameters to set secrets in YAML format:

transform:    

    ...

    secrets:

        name: my-ssh-key

        env_var: MY_KEY

        key: "mykey"

Here is how to do the same in JSON format:

"transform": {

    ...

    "secrets": {

        "name": "my-ssh-key",

        "env_var": "MY_KEY",

        "key": "my_key"

    }

Next, let's learn about image_pull_secrets.

image_pull_secrets

The image_pull_secrets parameter enables you to configure your Pachyderm pipeline to pull images from a private Docker registry. To specify this parameter, you need to create a Kubernetes secret with a Docker config, as described in the Kubernetes documentation at https://kubernetes.io/docs/concepts/containers/images/#creating-a-secret-with-a-docker-config, and then specify the secret in the pipeline specification under the image_pull_secrets parameter. You will need to use a full path to the Docker image for the pipeline to pull the image correctly.

The following code shows the syntax you can use in the image_pull_secrets parameter to enable the pipeline to pull images from a private Docker registry in YAML format:

transform:    

    ...

    image: my-private-docker-registry.com/my-project/my-image:latest

    image_pull_secrets:

      - my-secret

This is how you would write the same in JSON format:

"transform": {

    ...

    "image": "my-private-docker-registry.com/my-project/my-image",        

    "image_pull_secrets": ["my-secret"]

The next parameter that we will review is accept_return_code.

accept_return_code

The accept_return_code parameter enables you to specify an array of integers that define error codes that your pipeline will still be considered successful with. You can use this functionality in cases where you want your code to succeed, even if some part of it failed. This parameter is similar to the err_cmd functionality.

The following code shows the syntax you can use in the accept_return_code parameter to specify error codes in YAML format:

transform:    

    ...

    accept_return_code:

    - 256

Here is the same example in JSON format:

"transform": {

    ...

    "accept_return_code": [256]

The next parameter we will look at is debug.

debug

The debug parameter enables you to set the verbosity of the pipeline's logging output. Basic logging is enabled by default, but if you'd like to include more detailed messaging, set this parameter to true. By default, this parameter is set to false.

Here is how you can enable debug logging for your pipeline in YAML format:

transform:    

    ...

    debug: true

This is how you would enable debug logging for your pipeline in JSON format:

"transform": {

    ...

    "debug": true

Next, let's learn how to use the user parameter.

user

The user parameter enables you to define a user and a group that runs the container's code. This parameter is similar to the Docker USER directive, and you can also define it through Dockerfile. By default, Pachyderm checks what's in your Dockerfile first and sets this value for the user parameter in the pipeline specification. If nothing is specified in your Dockerfile and the pipeline specification, the default parameter is used, which is root. The only time that you must explicitly specify a user in the pipeline specification is when you deploy Pachyderm with the --no-expose-docker-socket parameter.

You can read more about the Docker USER parameter at https://docs.docker.com/engine/reference/builder/#user.

Here is how you can specify user in YAML format:

transform:    

    ...

    user: test-user

Here is how you can specify user in JSON format:

"transform": {

    ...

    "user": "test-user"

Now, let's learn about the working_dir parameter.

working_dir

The working_dir parameter enables you to specify a working directory for your pipeline container. This parameter is similar to the Docker WORKDIR directive, and you can also define it through Dockerfile. By default, Pachyderm checks what's in your Dockerfile first and sets this value for the working_dir parameter in the pipeline specification. If nothing is specified in Dockerfile and the pipeline specification, the default parameter is used, which is the root directory (/) or the directory that the Docker image inherits from the base image. The only time that you must explicitly specify a working directory in the pipeline specification is when you deploy Pachyderm with the --no-expose-docker-socket parameter.

You can read more about the Docker Workdir parameter at https://docs.docker.com/engine/reference/builder/#workdir.

Here is how you can specify workdir in YAML format:

transform:    

    ...

    working_dir: /usr/src/test

The same parameter in JSON format would look like this:

"transform": {

    ...

    "working_dir": "/usr/src/test"

Next, we'll look at the dockerfile parameter.

dockerfile

The dockerfile parameter enables you to specify a path to the location of the Dockerfile for your pipeline. This is useful when you use Pachyderm's pachctl update-pipeline –build -f <pipeline-spec> to build new Docker images for your pipeline. By default, Pachyderm will look for a Dockerfile in the same directory as the pipeline specification. But with the dockerfile parameter, you can set any path for it.

Here is how you can specify a path to Dockerfile in YAML format:

transform:    

    ...

    dockerfile: /path/to/dockerfile

You can do the same in JSON format like this:

"transform": {

    ...

    "dockerfile": "/path/to/dockerfile "

In this section, we learned about all the parameters that you can specify for your transformation code. In the next section, we will review how to control pipeline worker performance and assign resource limits to optimize your pipeline.  

Optimizing your pipeline

This section will walk you through the pipeline specification parameters that may help you optimize your pipeline to perform better. Because Pachyderm runs on top of Kubernetes, it is a highly scalable system that can help you use your underlying hardware resources wisely.

One of the biggest advantages of Pachyderm is that you can specify resources for each pipeline individually, as well as defining how many workers your pipeline will spin off for each run and what their behavior will be when they are idle and waiting for new work to come.  

If you are just testing Pachyderm to understand whether or not it would work for your use case, the optimization parameters may not be as important. But if you are working on implementing an enterprise-level data science platform with multiple pipelines and massive amounts of data being injected into Pachyderm, knowing how to optimize your pipeline becomes a priority.

You must understand the concept of Pachyderm datums before you proceed. Datums play a major role in pipeline scalability. If you have not read Chapter 2, Pachyderm Basics, yet, you may want to read it before you continue.

parallelism_spec

parallelism_spec defines the number of workers to spin off for your pipeline. You can specify either a coefficient or constant policy. By default, Pachyderm deploys one worker per pipeline with a constant policy of 1.

The coefficient policy means that Pachyderm will create several workers proportional to the specified coefficient. For example, if you have 50 nodes in your Kubernetes cluster and set the coefficient policy to 1, Pachyderm will use all 50 nodes for this cluster. If you use the coefficient policy, your pipeline needs access to the Kubernetes administrative nodes. If you are running Pachyderm on a hosted version of Kubernetes, such as on the AWS or GKE platform, you may not have access to these, and the pipeline will constantly restart. In that case, you will have to use the constant policy instead.

The constant policy enables you to specify the exact number of worker nodes for your pipeline, such as 3, 25, or 100. These workers will run in this pipeline infinitely. However, if you want your cluster to spin them down when idle, you can set the standby:true parameter so that your cluster resizes dynamically based on the workload.

The following code shows the syntax you can use in the parallelism_spec parameter to specify the coefficient policy in YAML format:

parallelism_spec:

  coefficient: '1'

The following code shows the syntax you can use in the parallelism_spec parameter to specify the coefficient policy in JSON format:

"parallelism_spec": {

    "coefficient": "1"

  },

This is how you can define the parallelism_spec parameter to specify the constant policy in YAML format:

parallelism_spec: until_success

The following code shows how to use the parallelism_spec parameter to specify the constant policy in JSON format:

"parallelism_spec": "until_success"

Next, let's learn how to use reprocess_spec.

reprocess_spec

reprocess_spec enables you to force your pipeline to reprocess all datums. By default, Pachyderm skips reprocessing successful datums, but if your pipeline interacts with an external application or system, you may want to reprocess all the datums. This behavior protects the pipeline from connection and other errors.

The following is the syntax you can use in reprocess_spec in YAML format:

reprocess_spec:

  constant: '1'

The following is the syntax you can use in reprocess_spec in JSON format:

"parallelism_spec": {

    "constant": "1"

}

Next, we'll learn how to use cache_size.

cache_size

The cache_size parameter enables you to define the amount of cache memory for the user and storage container. Pachyderm pre-downloads the data before processing it and increasing cache_size may help increase the download speed. The default value is 64M, and you can increase this as needed to cache your datums. This is a fine-tuning parameter and should only be used once you have optimized your pipeline through glob and parallelism_spec.

The following is an example of the syntax you can use in the cache_size parameter to increase the cache's size in YAML format:

cache_size: 1G

In JSON format, the same parameter would look like this:

"cache_size": "1G",

Now, let's review the max_queue_size parameter.

max_queue_size

The max_queue_size parameter defines how many datums the pipeline can download at the same time. You can use max_queue_size to make the pipeline pre-download the next datum while the other datums are being processed. By default, Pachyderm sets max_queue_size to 1, meaning that only one datum can be downloaded at a time. This is a fine-tuning parameter that can improve the download speed of datums into the worker if the download time is significantly longer than the processing time. However, you should only adjust this parameter once you have configured the correct glob and paralellism_spec.

The following code shows how to use the max_queue_size parameter to increase the cache's size in YAML format:

max_queue_size: 5

The same key-value pair in JSON format looks as follows:

"max_queue_size": "5",

Next, we'll learn about chunk_spec.

chunk_spec

The chunk_spec parameter defines how many datums to send to each worker for processing. By default, this parameter is set to 1. A chunk of data can be set to number (number of datums) or size_bytes. For example, if you set chunk_spec in number (number of datums) to 3, each worker will get a chunk of three datums at a time.

With size_bytes, you can make evenly sized datums if the runtime of a datum is proportional to its size. If this is not the case, use the number parameter instead.

The following code shows how to set number in the chunk_spec parameter to make the workers process this number of datums at a time in YAML format:

chunk_spec:

  number: '10'

Here's how you can set number in the chunk_spec parameter to make the workers process this number of datums at a time in JSON format:

"chunk_spec": {

    "number": "10"

  },

The following code shows how to set size_bytes ( of datums) in the chunk_spec parameter to make the workers process this number of datums at a time in YAML format:

chunk_spec:

  size_bytes: '1210'

If you prefer to write in JSON format, setting size_bytes in the chunk_spec parameter to make the workers process this number of datums at a time would look like this:

"chunk_spec": {

    "size_bytes": "1210"

  },

Next, we will learn how to set resource limits.

resource_limits

The resource_limits parameter enables you to restrict the amount of Central Processing Unit (CPU), Graphics Processing Unit (GPU), and memory resources that your pipeline worker can use. You can specify type for your GPU resources. The pipeline worker cannot use more resources than the limit you specify. The resource_request parameter is similar, but the worker can go over the requested amount of resources if they are available. You can read more about resource_limits and resource_requests in the Kubernetes documentation at https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits.

The following code shows how to set the resource_limits parameter to limit Pachyderm worker resources in YAML format:

resource_limits:

  cpu: 1

  gpu:

    type: nvidia.com/gpu

    number: 1

    memory: 16G

The following code shows how to set the resource_limits parameter to limit Pachyderm worker resources in JSON format:

"resource_limits": {

    "cpu": 1,

    "gpu": {

        "type": "nvidia.com/gpu",

        "number": 1,

    "memory": "16G"

    }

  },

If you need to set a specific flavor of cloud resource, such as TPU in Google Kubernetes Engine, you can do so by configuring pod_patch. See the upcoming pod_patch section for more information.

resource_requests

The resource_requests parameter specifies the number of resources that a pipeline worker requests to process a unit of work. Unlike the resource_limits parameter, if more resources are available, the worker can use them. The syntax for this parameter is the same as for resource_limits.

sidecar_resource_limits

This parameter is similar to resource_limits and defines the resource for the pipeline sidecar container. For some example syntax, see the resource_limits section.

scheduling_spec

The scheduling_spec parameter enables you to specify which Pods to run your pipeline code on based on a specified node_selector or priority_class. node_selectore enables you to specify a specific group of nodes that have the same label, called nodeSelector, while priority_class enables you to schedule the pipeline on a group of nodes that matches a Kubernetes PriorityClass. The scheduling_spec parameter is typically used to schedule pipelines on specific workers because of the resources they provide. For more information about these properties, see https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector and https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass.

The following code shows how to define nodeSelector in YAML format:

scheduling_spec:

  node_selector:

    kubernetes.io/my-hostname: my-node

The following code shows how to define nodeSelector in JSON format:

"scheduling_spec": {

      "node_selector": {"kubernetes.io/my-hostname": "my-node"

      }

    }, 

To define the PriorityClass parameter in YAML format, use the following code:

scheduling_spec:

  priority_class: high-priority

    kubernetes.io/my-hostname: my-node

Or, if you prefer to write in JSON format, set PriorityClass like this:

"scheduling_spec": {

      "priority_class": "high-priority"

      }

Next, we'll learn how to set a timeout for a job.

job_timeout

The job_timeout parameter enables you to set a timeout for a Pachyderm job run. This means that if your job does not finish within the specified period, it will fail. By default, this parameter is disabled. You can set it to your preferred time value.

The following code shows how to define a job_timeout in YAML format:

job_timeout: 10m

Here is the same example in JSON format:

"job_timeout": "10m"

Next, we'll learn about datum_timeout.

datum_timeout

The datum_timeout timeout is similar to job_timeout, except that the timeout is set at the datum level of granularity. The syntax is the same.

datum_tries

The datum_tries parameter defines how many times Pachyderm will try to rerun a pipeline on a failed datum. By default, this parameter is set to 3. If you want your pipeline to run only once and not try to process the failed datums again, set this value to 1.

The following code shows how to define the job_tries parameter in YAML format:

datum_tries: 1

If you prefer to write in JSON format, you can use the following code instead:

"datum_tries": "1"

In this section, we learned how to achieve optimal performance by fine-tuning our pipeline specification. In the next section, you will learn how to configure some of the service parameters that assist in pipeline operations.

Exploring service parameters

Now, let's look at service parameters. Service parameters include the parameters that let you collect statistics about your data, as well as patch your pipeline's Kubernetes configuration.

enable_stats

The enable_stats parameter, as its name suggests, enables pipeline statistics logging. By default, this parameter is disabled. For debugging purposes, it is recommended that you set this parameter to true. Once you enable statistics collection, the statistics are saved in the stats folder. You cannot disable statistics collection.

The following code shows how to define the enable_stats parameter in YAML format:

enable_stats: true

The following code shows how to define the enable_stats parameter in JSON format:

"enable_stats": true,

Next, we'll learn about pod_patch.

pod_patch

The pod_patch parameter enables you to rewrite any field in your pipeline Pods. This can be useful for many things, but one example is mounting a volume in your pipeline. To create a pod_patch, you would typically use a JSON patch builder, convert it into a one-liner, and add it to your pipeline specification.

The following code shows how to define the pod_patch parameter in YAML format:

pod_patch: '[{"op": "add","path": "spec/initContainers/0/resources/limits/my-volume"}]'

The same in JSON format looks like this:

"pod_patch": "[{\"op\": \"add\",\"path\": \"spec/initContainers/0/resources/limits/my-volume\"}]"

This is all you need to know about service parameters. In the next section, we will look at some parameters that enable you to configure the output branch and write your pipeline results to external storage.

Exploring output parameters

Output parameters enable you to configure what happens to your processed data after the result lands in the output repository. You can set it up to be placed in an external S3 repository or configure an egress.

s3_out

The s3_out parameter enables your Pachyderm pipeline to write output to an S3 repository instead of the standard pfs/out. This parameter requires a Boolean value. To access the output repository, you would have to use an S3 protocol address, such as s3://<output-repo>. The output repository will still be eponymous to your pipeline's name.

The following code shows how to define an s3_out parameter in YAML format:

s3_out: true

Here's how to do the same in JSON format:

"s3_out": true

Now, let's learn about egress.

egress

The egress parameter enables you to specify an external location for your output data. Pachyderm supports Amazon S3 (the s3:// protocol), Google Cloud Storage (the gs:// protocol), and Azure Blob Storage (the wasb:// protocol).

The following code shows how to define an egress parameter in YAML format:

egress:

  URL: gs://mystorage/mydir

Here's the same example but in JSON format:

"egress": {

        "URL": "gs://mystorage/mydir"

  },

Next, let's learn about how to configure an output branch in Pachyderm.

output_branch

The output_branch parameter enables you to write your results into a branch in an output repository that is different from the default master branch. You may want to do this if you want to create an experiment or a development output that you don't want the downstream pipeline to pick up.

The following code shows how to define the output_branch parameter in YAML format:

output_branch: test

The following code shows how to define the output_branch parameter in JSON format:

"output_branch": "test",

This concludes our overview of the Pachyderm pipeline specification.

Summary

In this chapter, we learned about all the parameters you can specify in a Pachyderm pipeline, how to optimize it, and how to configure the transformation section. The pipeline specification is the most important configuration attribute of your pipeline as you will use it to create your pipeline. As you have learned, the pipeline specification provides a lot of flexibility regarding performance optimization. While it may be tricky to find the right parameters for your type of data right away, Pachyderm provides a lot of fine-tuning options that can help you achieve the best performance for your ML workflow.

In the next chapter, you will learn how to install Pachyderm on your local computer.

Further reading

To learn more about the topics that were covered in this chapter, take a look at the following resources:

Pachyderm Environment Variables: https://docs.pachyderm.com/latest/deploy-manage/deploy/environment-variables/

Kubernetes Concepts: https://kubernetes.io/docs/concepts/

Pachyderm Pipeline Specification: https://docs.pachyderm.com/latest/reference/pipeline_spec/

Section 2:Getting Started with Pachyderm

This section walks you through the process of installing and configuring Pachyderm on your computer or in a cloud environment and the process of creating your first pipeline, and then explains how to configure an end-to-end machine learning workflow.

This section comprises the following chapters:

Chapter 4, Installing Pachyderm Locally

Chapter 5, Installing Pachyderm on a Cloud Platform

Chapter 6, Creating Your First Pipeline

Chapter 7, Pachyderm Operations

Chapter 8, Creating an End-to-End Machine Learning Workflow

Chapter 9, Distributed Hyperparameter Tuning with Pachyderm

Chapter 4: Installing Pachyderm Locally

In the previous chapters, we learned about Pachyderm's architecture, the internals of the Pachyderm solution, and version control primitives such as repositories branches, and commits. We reviewed why reproducibility is essential and why it should be a part of a successful data science process. We also learned how to do this on all three major platforms – macOS, Linux, and Windows.

There are many ways and a variety of platforms that enable you to run your end-to-end Machine Learning (ML) workflows using Pachyderm. We will start with the most common and easy to configure local deployment method on your computer; then, in the following chapters, we will review the deployment process on cloud platforms.

This chapter will walk you through the process of installing Pachyderm locally so that you can get started quickly and test Pachyderm. This chapter will prepare you to run your first pipeline. We will provide an overview of the system requirements and guide you through the process of installing all the prerequisite software needed for Pachyderm to run smoothly.

In this chapter, we're going to cover the following main topics:

Installing the required tools

Installing minikube

Installing Docker Desktop

Installing the Pachyderm Command-Line Interface (CLI)

Enabling autocompletion for Pachyderm

Preparing the Kubernetes environment

Deploying Pachyderm

Accessing the Pachyderm dashboard

Deleting an existing Pachyderm deployment

Technical requirements

Whether you are on macOS, Windows, or Linux, you need to install the following tools:

Homebrew

The Kubernetes CLI; that is, via kubectl

Helm

minikube

Docker Desktop (as an alternative to minikube)

The Pachyderm CLI; that is, via pachctl

Windows Subsystem for Linux (WSL) for Windows installations

We will get into the specifics of installing and configuring these tools as we go through this chapter. If you already know how to do this, you can go ahead and set them up now.

Installing the required tools

In this section, we will cover how to install the system tools that we will use to prepare our environment before installing Pachyderm.

Installing Homebrew (macOS only)

While Linux distributions have many package management options, there is no default package manager for macOS users. Homebrew (brew) fills this gap and provides a great solution to easily install and manage software from the macOS Terminal and Linux shell as an alternative to apt, yum, or flatpak, which are available in Linux distributions.

Homebrew uses Git to download its updates. In Homebrew, packages are installed based on definitions known as Formulae. Homebrew installs software packages to the Cellar, which is located under the /user/local/Cellar directory. Another term you will hear often is Tap. Tap is a Git repository of Formulae.

In this chapter, we will frequently use brew to install various software packages on macOS. Therefore, you need to install it if you are using macOS. The same brew commands we will use in this chapter run on Linux as well, but we will keep the use of brew for Linux optional:

Execute the following command to install Homebrew on your computer:

$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Execute the following command to verify that Homebrew has been installed, as well as to list the available commands:

$ brew commands

The following screenshot shows the system's output:

Figure 4.1 – List of brew commands

Let's learn about some of the useful Homebrew commands you may need in the future. You can update Homebrew itself by executing the following command:

$ brew update

Execute the following command to find any outdated formulae:

$ brew outdated

Execute the following command to upgrade all the outdated formulae:

$ brew upgrade

Now that you have installed the Homebrew package manager on your computer, let's install kubectl.

Installing Windows Subsystem for Linux (for Windows only)

WSL is a tool that enables Windows users to run Linux commands and utilities natively in Windows. If you are using Windows, you can install WSL on your machine by following these steps:

Open PowerShell or the Windows Command Prompt.

Install WSL by running the following command:

wsl --install

Important note

If you are on Windows, run all the Linux and Pachyderm commands described in this book from WSL.

For more information, see the official Microsoft Windows documentation at https://docs.microsoft.com/en-us/windows/wsl/install.

Installing the Kubernetes command-line tool

Before you create your first Kubernetes cluster, you need to install the Kubernetes command-line tool, kubectl, to execute commands against the cluster. Now, let's learn how to install kubectl on a computer.

See the official Kubernetes documentation for more information: https://kubernetes.io/docs/home/.

Follow these steps:

Execute the following commands to download and install kubectl on your computer:

If you are using Linux, run the following command:

$ curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl

$ chmod +x ./kubectl && sudo mv ./kubectl /usr/local/bin/kubectl

If you are on macOS (Intel), run the following command:

$ curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl

$ chmod +x ./kubectl && sudo mv ./kubectl /usr/local/bin/kubectl

If you are on Windows, the following command will do the trick:

curl -LO https://dl.k8s.io/release/v1.22.0/bin/windows/amd64/kubectl.exe

Verify the version you are using and make sure that kubectl is installed by executing the following command:

$ kubectl version --short --client

Here is an example of the system's output:

Client Version: v1.22.3

To be able to perform the following commands, the kubectl version must be v1.19 or later.

Now that you have kubectl installed on your computer to execute commands against your Kubernetes cluster, let's install Helm.

Installing Helm v3

Helm is a popular package manager for Kubernetes clusters. Before you deploy Pachyderm by using its Helm chart, you need to install the Helm binary on your environment to be able to manage the life cycle of your Helm chart. Follow these steps to install Helm on your computer:

Execute the following command to download and install Helm on your computer:

$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/

helm/helm/master/scripts/get-helm-3

$ chmod 700 get_helm.sh

$ ./get_helm.sh

Verify the version you are using and make sure Helm is installed by executing the following command:

$ helm version --short

Here is an example of the system's output:

V3.7.1+g1d11fcb

Next, you must install the necessary tools to prepare your local Kubernetes cluster environment before you can deploy Pachyderm. If you are familiar with containers in Linux, you must be familiar with these tools. If you are using Linux as your local machine, follow the instructions provided in the Installing minikube section to prepare your environment. If you are using macOS, follow the instructions provided in the Installing Docker Desktop section. Using Docker Desktop is recommended due to its simplicity.

Installing minikube

Minikube is a popular cross-platform and lightweight Kubernetes implementation that helps users quickly create a single-node local Kubernetes cluster. minikube supports multiple runtimes, including CRI-O, container, and Docker. It can be deployed as a Virtual Machine (VM), a container, or on bare metal. Since Pachyderm supports the Docker runtime only, we will cover how to use the Docker container runtime and deploy it as a container. For additional configuration details, you can refer to the official Docker documentation at https://minikube.sigs.k8s.io/docs/start/. Let's install the latest version of minikube:

Execute the following commands to install minikube on your computer.

If are using Linux, run the following command:

$ curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64

$ sudo install minikube-linux-amd64 /usr/local/bin/minikube

If you are on Windows (Chocolatey package manager is required), run the following command:

choco install minikube

Verify the minikube version you are using and make sure that minikube has been installed by executing the following command:

$ minikube version

The following is an example of the command's response:

minikube version: v1.22.0

commit: a03fbcf166e6f74ef224d4a63be4277d017bb62e

Now that you have installed minikube installed, let's install Docker Desktop.

Installing Docker Desktop

Docker simplifies developing, delivering, and running applications by separating applications from the infrastructure and its dependencies. Pachyderm supports the Docker container runtime only; therefore, Docker tools must be installed before you deploy Pachyderm.

Docker runs as a native application using the macOS sandbox security model and installs all Docker tools on your macOS, including Docker Engine, the CLI, Docker Compose, Credential Helper, Notary, and Kubernetes.

If you do not have Docker Desktop already installed, you can follow the instructions provided in the next section to install it. Otherwise, you can skip to the Preparing your Kubernetes environment section. You can also refer to the official Docker documentation at https://docs.docker.com/get-docker/.

Installing Docker Desktop for macOS

Follow these steps to install Docker Desktop on macOS. The latest version of Docker is supported on the last three versions of macOS. If your macOS version is older than the last three versions, you need to upgrade it to the latest version of macOS:

Visit the Docker Hub download link at https://hub.docker.com/editions/community/docker-ce-desktop-mac/ to download the Docker Desktop installer.

Click on the Get Docker button to download the Docker.dmg file to your workstation.

Once the download has completed, double-click on the Docker.dmg file to open the image and drag the Docker icon from the window to the Applications folder to complete the installation:

Figure 4.2 – Docker Desktop installation screen on macOS

In the Applications folder, double-click on the Docker icon to start Docker Desktop.

Confirm that you have privileged access to install Docker components on your workstation.

If you are using Docker for the first time, follow the quick tutorial. Otherwise, click Skip Tutorial to start Docker:

Figure 4.3 – Docker Desktop graphical user interface

Installing Docker Desktop on Windows

Install Docker Desktop on your Windows machine by following these steps:

Go to https://docs.docker.com/desktop/windows/install/.

Click Docker Desktop for Windows.

Click Docker Desktop Installer.exe.

Follow the interactive prompts to install Docker Desktop.

On the Configuration page, make sure that you select Install required Windows components for WSL 2.

When you're done, click Close.

Start Docker Desktop by finding it in Windows Search and accept the terms and conditions. Docker will start automatically after that.

Now that you have installed Docker Desktop on your machine, let's install the Pachyderm CLI, called pachctl.

Installing the Pachyderm command-line interface

The Pachyderm CLI, pachctl, is used to deploy and interact with Pachyderm clusters. Follow these steps to install pachctl:

Let's get the latest release version tag of pachctl and keep it in a variable called PACHYDERMVERSION:

$ PACHYDERMVERSION=$(curl --silent "https://api.github.com/repos/pachyderm/pachyderm/releases/latest" | grep '"tag_name":' | \

sed -E 's/.*"v([^"]+)".*/\1/')

Execute the following commands to install pachctl on your computer.

If you are using macOS, run the following command:

$ brew tap pachyderm/tap && brew install pachyderm/tap/pachctl@${PACHYDERMVERSION}

If you are using Debian Linux or WSL on Windows 10, run the following command:

$ curl -o /tmp/pachctl.deb -L https://github.com/pachyderm/pachyderm/releases/download/v${PACHYDERMVERSION}/pachctl_${PACHYDERMVERSION}_amd64.deb && sudo dpkg -i /tmp/pachctl.deb

Execute the following command to verify that you have installed pachctl:

$ pachctl version --client-only

The following is an example of the system's output:

COMPONENT           VERSION

pachctl             2.0.1

With that, you have installed the prerequisites to run Pachyderm locally. Now, let's prepare our cluster and deploy Pachyderm on our local Kubernetes cluster.

Enabling autocompletion for Pachyderm

Autocompletion is a functionality that's offered by Unix shell flavors to autofill parameters using the CLI. Depending on the type of shell that's used in your system, the autocompletion feature suggests or autocompletes the partially typed commands as you type, sometimes by pressing the Tab key. Pachyderm supports autocompletion for Bourne Again Shell (bash) and Z shell (zsh), an extended Bourne shell. bash and zsh are the most common Unix command-line interpreters that are used on macOS and Linux. In this section, you will learn how to enable the Pachyderm autocompletion feature and the parameters that are available from the pachctl command.

If you don't know which shell you are using, type the following command to find out:

$ echo "$SHELL"

If you are using bash, the output of the preceding command should look as follows:

/bin/bash

If you are using zsh, the output of the preceding command should look as follows:

/bin/zsh

Since we now know which shell we are using, we can install Pachyderm autocompletion.

Enabling Pachyderm autocompletion for bash

Follow these steps to enable Pachyderm autocompletion on your computer:

Execute the following commands to enable bash-completion.

If you are using macOS or Linux with Homebrew, use the following command:

$ brew install bash-completion

If you are on Ubuntu Linux, use the following command:

$ sudo apt install bash-completion

If you are using RHEL or CentOS Linux, use the following command:

$ sudo yum install bash-completion bash-completion-extras

Execute the following command to verify that bash-completion is enabled on your computer.

If you are on macOS, run the following command:

$ brew info bash-completion

If you are using Linux, run the following command:

$ complete -p

Confirm that the path for bash-completion is pointing to the correct directory. Then, execute either of the following commands to enable Pachyderm pachctl autocompletion:

If you are on macOS, run the following command:

$ pachctl completion bash --install --path /usr/local/etc/bash_completion.d/pachctl

If you are using Linux, run the following command:

$ pachctl completion bash --install --path /usr/share/bash-completion/completions/pachctl

With that, Pachyderm's command-line autocompletion has been enabled in your bash shell.

Enabling Pachyderm autocompletion for zsh

Z shell, or zsh, is an improved interactive login shell with many advanced features. The default interactive shell in Apple laptops was changed to zsh with macOS Catalina. Follow these steps to enable Pachyderm autocompletion on your computer:

Important note

If you do not wish to enable autocompletion, you can try using pachctl shell instead. To enable this feature, type pachctl shell.

Execute the following commands to enable zsh autocompletion.

If you are using macOS or Linux with Homebrew, use the following command:

$ brew install zsh-completion

If you are on Linux, visit the https://github.com/zsh-users/zsh-completions page and follow the instructions for your Linux distribution to enable zsh completion. As an example, for Ubuntu 19.10, this would look as follows:

$ echo 'deb http://download.opensuse.org/repositories/shells:/zsh-users:/zsh-completions/xUbuntu_19.10/ /' | sudo tee /etc/apt/sources.list.d/shells:zsh-users:zsh-completions.list

$ curl -fsSL https://download.opensuse.org/repositories/shells:zsh-users:zsh-completions/xUbuntu_19.10/Release.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/shells_zsh-users_zsh-completions.gpg > /dev/null

$ sudo apt update && sudo apt install zsh-completions

Execute the following commandss to verify that zsh autocompletion is enabled on your computer.

If you are on macOS, run the following command:

$ brew info zsh-completions

If you are using Linux, run the following command:

$ complete -p

Confirm that the path for zsh autocompletion is pointing to the correct directory. Then, execute either of the following commands to enable Pachyderm pachctl autocompletion.

On macOS, run the following command:

$ pachctl completion zsh --install --path /usr/local/share/zsh-completions/_pachctl

If you are using Linux, run the following command:

$ pachctl completion zsh --install --path /home/linuxbrew/.linuxbrew/share/zsh-completions/_pachctl

With that, Pachyderm command-line autocompletion is now enabled in your zsh shell. Next, let's prepare the Kubernetes environment.

Preparing the Kubernetes environment

In this section, you will provision a Kubernetes cluster by using the preferred tools that you deployed in the Installing the required tools section.

Enabling Kubernetes on Docker Desktop

Follow these steps to enable Kubernetes if you're using Docker Desktop as your container platform to deploy your Kubernetes cluster on both Windows and macOS:

Open the Docker UI.

At the top-right corner of the Docker UI, click the Settings icon.

Switch to the Kubernetes settings panel and click the Enable Kubernetes button to start a single-node Kubernetes cluster with Docker Desktop. Apply these settings by clicking the Apply & Restart button:

Figure 4.4 – Enabling Kubernetes in Docker Desktop

Open a Terminal window and confirm that Kubernetes is running by executing the following command:

$ kubectl get node

The following is an example of the system's response:

NAME           STATUS ROLES                AGE  VERSION

docker-desktop Ready  control-plane,master 7m9s v1.21.5

With that, you have a single-node Kubernetes cluster configured on Docker Desktop. Now, we are ready to deploy Pachyderm on our local Kubernetes environment.

Enabling Kubernetes using minikube

Follow these steps to run Kubernetes locally when using minikube:

Make Docker the default driver for minikube:

$ minikube config set driver docker

Start a Kubernetes cluster:

$ minikube start

Execute the following command to verify that your Kubernetes cluster is ready:

$ kubectl get node

NAME       STATUS   ROLES                  AGE   VERSION

minikube   Ready    control-plane,master   29m   v1.20.2

With that, your Kubernetes cluster has been configured using minikube. Now, we are ready to deploy Pachyderm on our local Kubernetes environment.

Deploying Pachyderm

When running Pachyderm in production, it is recommended to start in an environment where resources can scale up to handle the computational needs of larger pipelines. Pachyderm can be installed on any Kubernetes cluster, including managed Kubernetes services provided by AWS, Google Cloud, Microsoft Azure, IBM Cloud, and OpenShift, as well as locally on your workstation. In this section, we are going to focus on a smaller test deployment; therefore, a local cluster is good enough to get started.

Pachyderm provides sample Helm charts to help you deploy Pachyderm to all major cloud platforms. You can read more about Helm Charts in Chapter 2, Pachyderm Basics. Because Helm charts are flexible, you can pick the components that you want to install. For example, you can install the Pachyderm in-browser interface called the Console.

The Pachyderm Console is the Pachyderm user interface and provides a birds-eye view of your pipelines through the Direct Acyclic Graph (DAG), as well as other useful features.

Some components, such as the Pachyderm Console, require an Enterprise license but are also available for testing with a free trial license for 30 days. You can request a free trial license at https://www.pachyderm.com/trial/.

Follow these steps to install Pachyderm on your local Kubernetes cluster:

Deploying Pachyderm consists of multiple resources. The helm command helps manage complex Kubernetes applications' life cycles and creates all the necessary components at once using a single command. You can learn more about the options for Helm charts in Chapter 5, Installing Pachyderm on a Cloud Platform. For now, let's execute the following command to add Pachyderm Helm chart repositories to your local repository:

$ helm repo add pach https://helm.pachyderm.com  

Execute the following command to get the latest chart information from the chart repository:

$ helm repo update

Execute the following command to deploy the latest version of Pachyderm on your cluster without the console:

$ helm install pachd pach/pachyderm --set deployTarget=LOCAL

If you have an Enterprise key and you would like to deploy it with Pachyderm's console user interface, create a file called license.txt and paste your Enterprise token into that file. Then, run the following commands:

$ helm install pachd pach/pachyderm --set deployTarget=LOCAL --set pachd.enterpriseLicenseKey=$(cat license.txt) --set console.enabled=true  

Once the Console has been deployed successfully, follow the instructions provided in the Accessing the Pachyderm Console section to access the Console.

The preceding commands return the following output:

Figure 4.5 – Pachyderm Helm chart getting deployed on Kubernetes

You can inspect the values.yaml file in the Helm Chart repository (https://github.com/pachyderm/pachyderm/tree/master/etc/helm/pachyderm) to learn more about the Kubernetes objects that are created by deploying the Chart. The Pachyderm Helm chart creates Kubernetes service accounts, Services, Deployments, PostgreSQL, and etcd instances, all of which are needed to run Pachyderm.

A Kubernetes Deployment is a controller that rolls out a ReplicaSet of Pods based on the requirements defined in your manifest file. A ReplicaSet is a group of the same service instances. Execute the following command to verify the state of the Deployments that have been created during the installation:

$ kubectl get deployments

The output of the preceding command should look as follows:

Figure 4.6 – List of Pachyderm Deployment objects

Execute the following command to verify that the installation was successful and view the Pods that were created as part of the Deployments:

$ kubectl get pods

The output of the preceding command should look as follows:

Figure 4.7 – List of Pachyderm Pods

Execute the following commands to connect to your new Pachyderm instance:

pachctl config import-kube local –overwrite

pachctl config set active-context local

pachctl port-forward

Go to your browser and point it to http://localhost:4000.

Authenticate with a mock user called admin and use password as your password.

Go back to your terminal and enable authentication:

pachctl auth activate

You'll be prompted to log into the UI again. Log in with the mock user called admin and use password as your password.

Execute the following command to verify that Pachyderm has been installed successfully:

pachctl version

The output of the preceding command should look as follows:

COMPONENT           VERSION

pachctl             2.0.1

pachd               2.0.1

Now that we have installed Pachyderm on our cluster, we are ready to create our first pipeline.

Accessing the Pachyderm Console

If you have installed the Console with your Pachyderm cluster, you can access it and view your pipelines, repositories, and other Pachyderm objects in the UI. The Pachyderm Console is available as a free trial for 30 days. Follow these steps to access the Pachyderm Console:

Execute the following command to verify the status of your Enterprise activation:

pachctl enterprise get-state

The output of the preceding command should look as follows:

Pachyderm Enterprise token state: ACTIVE

Expiration: 2022-02-02 22:35:21 +0000 UTC

If port-forwarding is not running, execute the following command to start it. This step will forward the Pachyderm Console service on port 4000 to your local workstation on port 4000:

pachctl port-forward

In a web browser, open http://localhost:4000.

Use the default credentials of admin and password on the login screen, as follows:

Figure 4.8 – Login screen of the Pachyderm Console

Upon clicking Login to access the Pachyderm Console, click the View Project button to view your repositories and pipelines:

Figure 4.9 – Pachyderm dashboard

Because we have not created any Pachyderm objects, this page is empty.

Now that you have learned how to access the Pachyderm Console, you are ready to create your first pipeline in Pachyderm.

Deleting an existing Pachyderm deployment

Only perform the steps in this section if you want to delete your cluster. If you want to continue working on the examples in other chapters, then please skip this section.

If you need to delete your deployment and start afresh, you need to wipe out your environment and start over again from the steps provided in the Preparing the Kubernetes environment section. When you delete an existing Pachyderm deployment, all the components, except for the Helm repository and pachctl, are removed from your machine.

Follow these steps to delete your existing Pachyderm deployment:

If you have used a different name for your Helm instance, then execute the following command to find the Pachyderm instance name that you deployed using the Helm chart:

$ helm ls | grep pachyderm

The output of the preceding command should look as follows:

pachd default 1 2021-11-08 21:33:44 deployed Pachyderm-2.0.1 2.0.1

Execute the following command using your Pachyderm instance name to remove the Pachyderm components from your cluster:

$ helm uninstall pachd

If you're using minikube, remove the entire Kubernetes cluster and redeploy before you deploy Pachyderm again:

$ minikube stop

$ minikube delete

With that, you have completely removed Pachyderm and the local Kubernetes cluster from your computer.

Summary

In this chapter, we learned about the software prerequisites for getting Pachyderm up and running on your local computer for testing purposes.

We gained basic knowledge about minikube and Docker Desktop and learned how to install them on our local machine. We also learned how to install the Pachyderm CLI and enable autocompletion on different operating systems.

We then installed Helm and the Pachyderm Helm repository on our system. We learned about Helm charts and how to obtain a free trial Pachyderm license.

We deployed a single-node, local Kubernetes cluster by using the most popular options available based on our desktop operating system. Finally, we deployed Pachyderm and learned how to access the Pachyderm Console.

We also learned how to do so on all three major platforms – macOS, Linux, and Windows.

In the next chapter, we will learn about how to install Pachyderm via the cloud and explain the software requirements needed to run a Pachyderm cluster in production. We will also learn about Pachyderm Hub, the Software-as-a-Service (SaaS) version of Pachyderm that is great for both testing and production environments.

Further reading

Please refer to the following links for more information about the topics that were covered in this chapter:

Official Homebrew Documentation: https://docs.brew.sh/

Kubectl Cheat Sheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/

Minikube Handbook: https://minikube.sigs.k8s.io/docs/handbook/

The Official Docker Desktop Documentation: https://docs.docker.com/desktop/

Kubernetes in Production Best Practices: https://www.packtpub.com/product/kubernetes-in-production-best-practices/9781800202450

Kubernetes and Docker – An Enterprise Guide: https://www.packtpub.com/product/kubernetes-and-docker-an-enterprise-guide/9781839213403

Official Pachyderm pachctl Reference Guide: https://docs.pachyderm.com/latest/reference/pachctl/pachctl/

Chapter 5: Installing Pachyderm on a Cloud Platform

In the previous chapter, you learned the process for installing Pachyderm locally to get started quickly and test Pachyderm on your computer.

Production use cases require additional compute resources and scalability that can be efficiently achieved using cloud platforms and managed Kubernetes platform services provided by the popular cloud vendors. Pachyderm can run on a Kubernetes cluster, irrespective of whether it is deployed manually on cloud instances or as a managed Kubernetes service. We will discuss the most popular and easy-to-configure methods on cloud providers.

This chapter walks you through the cloud-based installation of Pachyderm and explains the software requirements needed to run a Pachyderm cluster in production. This chapter will cover the installation on the following most popular cloud platforms: Amazon Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE), and Microsoft Azure Kubernetes Service (AKS).

In this chapter, we're going to cover the following main topics:

Installing the required tools

Deploying Pachyderm on Amazon EKS

Deploying Pachyderm on GKE

Deploying Pachyderm on Microsoft AKS

Accessing the Pachyderm console

Technical requirements

If you are on macOS, verify that you have an up-to-date version of macOS. If you are using Linux, you must be on 64-bit versions of recent distributions of CentOS, Fedora, or Ubuntu. If you are on Windows, run all the commands described in this section from Windows Subsystem for Linux (WSL). You should have the following tools installed from the previous chapters:

Homebrew (macOS only)

The Kubernetes Command-Line Interface (CLI) – kubectl

Helm

Pachyderm CLI – pachctl

WSL (Windows only)

We will need to install the following tools:

The Amazon Web Services (AWS) CLI – aws

The Amazon Identity and Access Management (AWS IAM) authenticator for Kubernetes

The EKS command-line tool – eksctl

The Google Cloud SDK – gcloud

The Azure CLI – az

A JSON processor – jq

We will go into the specifics regarding the installation and configuration of these tools as we go through this chapter. If you already know how to do this, you can go ahead and set them up now.

Installing the required tools

In this section, we will cover the installation of the system tools that we will use to prepare our environment before deploying a Kubernetes cluster and installing Pachyderm on cloud platforms.

Installing the AWS Command Line Interface to manage AWS

The AWS Command Line Interface, aws-cli, is required to execute commands in your AWS account. For additional information, you can refer to the AWS Command Line Interface official documentation at https://docs.aws.amazon.com/cli/latest/userguide/. Let's install aws-cli on your computer:

Execute the following commands to install aws-cli version 2 on your computer:

If you are using macOS:

$ curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"

$ sudo installer -pkg AWSCLIV2.pkg -target

If you are on Linux (x86) or WSL on Windows:

$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

$ unzip awscliv2.zip

$ sudo ./aws/install

Execute the following command to verify the successful installation of the AWS CLI:

$ aws --version

The output of the preceding command should look as follows:

aws-cli/2.4.7 Python/3.8.8 Linux/5.11.0-41-generic exe/x86_64.ubuntu.20 prompt/off

Execute the following command to configure the AWS CLI and enter your AWS access key and secret access key when asked. The AWS CLI saves the information used here in a credentials file located under ~/.aws/credentials to be used when you run the command later:

$ aws configure

The output of the preceding command should look as follows:

AWS Access Key ID [None]: YOURACCESSKEYHERE

AWS Secret Access Key [None]: YOURSECRETACCESSKEYHERE

Default region name [None]: us-east-1

Default output format [None]: json

To use a different account than what you have configured in the credentials file, you can set the environment variable, which will be used until the end of your current shell session. Use the following variables temporarily to set a user configuration if needed:

$ export AWS_ACCESS_KEY_ID=YOURACCESSKEY2HERE

$ export AWS_SECRET_ACCESS_KEY=YOURSECRETACCESS2KEYHERE

$ export AWS_DEFAULT_REGION=us-west-2

Now that you have installed the AWS Command Line Interface on your computer, let's install the AWS IAM authenticator for Kubernetes.

Installing the AWS IAM authenticator for Kubernetes

Amazon EKS leverages AWS IAM to provide access to the Kubernetes clusters created through EKS. To be able to make the kubectl command work with Amazon EKS IAM roles, the Amazon IAM authenticator for Kubernetes needs to be installed. Let's install the IAM authenticator on your computer:

Execute the following commands to install aws-iam-authenticator:

If you are using macOS:

$ brew install aws-iam-authenticator

If you are using Linux (x86) or WSL on Windows:

$ curl -o aws-iam-authenticator https://amazon-eks.s3.us-west-2.amazonaws.com/1.19.6/2021-01-05/bin/linux/amd64/aws-iam-authenticator

$ chmod +x ./aws-iam-authenticator

$ mkdir -p $HOME/bin && cp ./aws-iam-authenticator $HOME/bin/aws-iam-authenticator && export PATH=$PATH:$HOME/bin

$ echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc

Verify its version and make sure that aws-iam-authenticator is installed:

$ aws-iam-authenticator version

The output of the preceding command should look as follows. To be able to perform the following recipes, the aws-iam-authenticator version should be 0.5.0 or later:

{"Version":"v0.5.0","Commit":"1cfe2a90f68381eacd7b6dcfa2 bf689e76eb8b4b"}

Now you have installed aws-iam-authenticator on your computer.

Installing eksctl to manage Amazon EKS

Amazon EKS is a managed Kubernetes service on Amazon EC2. To manage Amazon EKS over the terminal and execute commands, the official CLI for Amazon EKS, eksctl, is used. For additional information, you can refer to the AWS eksctl official documentation at https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html. Let's install eksctl on your computer:

Execute the following commands to install eksctl on your computer:

If you are using macOS:

$ brew tap weaveworks/tap

$ brew install weaveworks/tap/eksctl

If you are using Linux (x86) or WSL on Windows:

$ curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp

$ sudo mv /tmp/eksctl /usr/local/bin

Verify its version and make sure that eksctl is installed:

$ eksctl version

The output of the preceding command should look as follows:

0.77.0

To be able to perform the following steps, the eksctl version should be 0.77.0 or later. Now you have installed eksctl to manage Amazon EKS on your computer.

Installing the Google Cloud SDK to manage Google Cloud

The Google Cloud SDK, gcloud, is required to execute commands in your Google Cloud account. The following instructions assume that you have an active GCP account with billing enabled. If you don't have an account already, go to https://console.cloud.google.com and create an account. Let's install gcloud on your computer:

Execute the following commands to install gcloud on your computer:

If you are using macOS:

$ brew tap weaveworks/tap

$ brew install weaveworks/tap/eksctl

If you are using Linux (x86) or WSL on Windows:

$ curl https://sdk.cloud.google.com | bash

Restart your shell by executing the following command:

$ exec -l $SHELL

Verify its version and make sure gcloud is installed:

$ gcloud version

The output of the preceding command should look as follows. To be able to perform the following recipes, the Google Cloud SDK version should be 339.0.0 or later:

Google Cloud SDK 367.0.0

bq 2.0.72

core 2021.12.10

gsutil 5.5

Execute the following command to initialize the SDK and follow the instructions:

$ gcloud init

Execute the following command to set a default zone for future deployments. In our example, the compute zone is set to us-central1-a:

$ gcloud config set compute/zone us-central1-a

Now you have installed gcloud to manage GKE on your computer.

Installing the Azure CLI to manage Microsoft Azure

The Azure CLI, az, is required to execute commands in your Microsoft Azure account. The following instructions assume that you have an active Azure account with billing enabled. If you don't have an account already, go to https://portal.azure.com and create an account. Let's install the Azure CLI on your computer:

Execute the following commands to install gcloud on your computer:

If you are using macOS:

$ brew update && brew install azure-cli

$ brew install jq

If you are using Linux (x86) or WSL on Windows:

$ curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

$ sudo apt-get install jq

Verify its version and make sure the Azure CLI is installed:

$ az version

The output of the preceding command should look as follows. To be able to perform the following steps, the Azure CLI version should be 2.0.1 or later:

{

  "azure-cli": "2.31.0",

  "azure-cli-core": "2.31.0",

  "azure-cli-telemetry": "1.0.6",

  "extensions": {}

}

Execute the following command to initialize the Azure CLI and follow the instructions:

$ az login

Execute the following command to create a resource group with a unique name and set a default zone for future deployments. In our example, the compute zone is set to centralus:

$ az group create --name="pachyderm-group" --location=centralus

Now you have installed the Azure CLI to manage AKS on your computer.

Deploying Pachyderm on Amazon EKS

Kubernetes is an open source container orchestration platform and, by itself, is a large topic to cover. In this section, we take the topic of containerization from a data scientist's perspective and will only focus on running our workload, Pachyderm, on the most common managed platforms available in the market. There are various ways and tools to provision and manage the life cycle of production-grade Kubernetes clusters on the AWS cloud platform, such as kOps, kubespray, k3s, Terraform, and others. For additional configuration details, you can refer to Kubernetes' official documentation at https://kubernetes.io/docs/setup/production-environment/. Let's learn the simplest way to get the services required by Pachyderm up and running on AWS's managed Kubernetes service, Amazon EKS.

Preparing an Amazon EKS cluster to run Pachyderm

Follow these steps to provision an Amazon EKS cluster using eksctl. Initially developed as a third-party open source tool, eksctl is now the official tool for deploying and managing EKS clusters via a CLI. You will need to have the AWS CLI and the AWS IAM authenticator for Kubernetes installed and their credentials configured. If you have a cluster, you can skip these instructions and jump to the Deploying Pachyderm on Amazon EKS section. Also, you can refer to the Amazon EKS official documentation at https://eksctl.io/introduction/:

Execute the following command to simply deploy an EKS cluster with default parameters. This command will generate a cluster with two m5.large worker nodes using the official AWS EKS Amazon Machine Image (AMI):

$ eksctl create cluster

The output of the preceding command should return output similar to this:

...

kubectl command should work with "/home/norton/.kube/config", try 'kubectl get nodes'

[✔]  EKS cluster "exciting-badger-1620255089" in "us-east-1" region is ready

Important note

To customize the EKS cluster configuration, you can pass additional parameters to eksctl as follows:

eksctl create cluster --name <name> --version <version> \

--nodegroup-name <name> --node-type <vm-flavor> \

--nodes <number-of-nodes> --nodes-min <min-number-nodes> \

--nodes-max <max-number-nodes> --node-ami auto

Verify the cluster deployment by executing the following command:

$ kubectl cluster-info && kubectl get nodes

The output of the preceding command should look as follows:

Kubernetes control plane is running at https://ABCD.gr7.us-east-1.eks.amazonaws.com

CoreDNS is running at https://ABCD.gr7.us-east-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

NAME                             STATUS   ROLES    AGE    VERSION

ip-192-168-17-133.ec2.internal   Ready    <none>   21m    v1.21.5-eks-bc4871b

ip-192-168-63-179.ec2.internal   Ready    <none>   21m    v1.21.5-eks-bc4871b

Now, your Amazon EKS cluster is provisioned and ready to deploy Pachyderm.

Creating an S3 object storage bucket

Pachyderm uses S3-compliant object storage to store data. Follow these steps to create an S3 object storage bucket:

Generate variables that will be used to create your S3 bucket and pass it to pachctl to store your Pachyderm data later. Make sure to use a unique bucket name. In our example, we will use s3.pachyderm as our bucket name with a capacity of 200 GB and located in the same region as our EKS cluster, us-east-1:

$ export S3_BUCKET_NAME=s3.pachyderm

$ export EBS_STORAGE_SIZE=200

$ export AWS_REGION=us-east-1

In order for Pachyderm to store pipeline data, a dedicated S3 bucket is required. Execute the following command to create an S3 bucket with the parameters defined by your variables:

$ aws s3api create-bucket --bucket ${S3_BUCKET_NAME} \

--region ${AWS_REGION}

Execute the following command to confirm the creation of the bucket:

$ aws s3api list-buckets --query 'Buckets[].Name'

The output of the preceding command should look as follows:

[

    "s3.pachyderm",

]

Now that we have an S3 bucket created, we are ready to deploy Pachyderm on Amazon EKS.

Deploying the cluster

When you start learning Pachyderm, it is recommended to run experiments in a small local cluster. We have previously covered the local deployment of Pachyderm in Chapter 4, Installing Pachyderm Locally. In this chapter, we focus on a scalable production-grade deployment of Pachyderm using IAM roles on Amazon EKS clusters.  

Follow these steps to install Pachyderm on your Amazon EKS cluster:

The AWS IAM role assigned to your EKS cluster needs to have access to the S3 bucket you created during the Creating an S3 object storage bucket section. As shown in the following screenshot, log in to your AWS Management Console and go to the EKS dashboard as follows:

Figure 5.1 – Amazon EKS Clusters dashboard

Click on the cluster. Locate the EC2 instance in the cluster. Find the IAM role on the Instance description page. Click on IAM Role:

Figure 5.2 – IAM role assigned to EC2 instances

Replace the <s3-bucket> placeholder below with your bucket name. Click on Add inline policy to create a new inline policy similar to the following code:

{

    "Statement": [

        {

            "Effect": "Allow",

            "Action": [

                "s3:PutObject",

                "s3:GetObject",

                "s3:ListBucket",

                "s3:DeleteObject"

            ],

            "Resource": [

                "arn:aws:s3:::<s3-bucket>",

                "arn:aws:s3:::*/*"

            ]

        }

    ]

}

Switch to the Trust relationships tab and click on the Edit trust relationship button. Confirm that the trust relationship is similar to the following, otherwise, make the changes and click on the Update Trust Policy button to update:

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Principal": {

        "Service": "ec2.amazonaws.com"

      },

      "Action": "sts:AssumeRole"

    }

  ]

}

Now, execute the following command to add the Pachyderm Helm Chart repos to your local repository:

$ helm repo add pach https://helm.pachyderm.com  

Execute the following command to get the latest Chart information from the Chart repository:

$ helm repo update

For a quick deployment, replace the Amazon S3 bucket name, access key ID, and secret key below and execute the command to deploy the latest version of Pachyderm on your cluster without the console:

$ helm install pachd pach/pachyderm \

--set deployTarget=AMAZON \

--set pachd.storage.amazon.bucket="AWS_S3_BUCKET" \

--set pachd.storage.amazon.id="AWS_ACCESS_KEY" \

--set pachd.storage.amazon.secret="AWS_SECRET" \

--set pachd.storage.amazon.region="us-east-1"

--set pachd.externalService.enabled=true

If you have an enterprise key and would like to deploy it with Pachyderm's console user interface, execute the following command:

$ helm install pachd pach/pachyderm \

--set deployTarget=AMAZON \

--set pachd.storage.amazon.bucket="AWS_S3_BUCKET" \

--set pachd.storage.amazon.id="AWS_ACCESS_KEY" \

--set pachd.storage.amazon.secret="AWS_SECRET" \

--set pachd.storage.amazon.region="us-east-1" \

--set pachd.enterpriseLicenseKey=$(cat license.txt) \

--set console.enabled=true

Once the console is deployed successfully, follow the instructions under the Accessing the Pachyderm console section to access the console.

The commands return the following output:

Figure 5.3 – Pachyderm Helm Chart getting deployed on Kubernetes

Optional: Customizing Installation Parameters

You can also download and customize the values.yaml file in the Helm Chart repository, https://github.com/pachyderm/pachyderm/tree/master/etc/helm/pachyderm, to further optimize the components needed to run Pachyderm.

Execute the following command to create a local copy of the values.yaml file:

$ wget https://raw.githubusercontent.com/pachyderm/pachyderm/master/etc/helm/pachyderm/values.yaml

Once customized, you can use the same YAML file and install your Helm Chart by executing the following command instead:

$ helm install pachyderm -f values.yaml pach/pachyderm

A Kubernetes Deployment is a controller that rolls out a ReplicaSet of Pods based on the requirements defined in your manifest file. A ReplicaSet is a group of instances of the same service. Execute the following command to verify the state of Deployments created during the installation:

$ kubectl get deployments

The output of the preceding command should look as follows:

Figure 5.4 – List of Pachyderm Deployment objects

Execute the following command to verify a successful installation and see the Pods created as part of the Deployments:

$ kubectl get pods

The output of the preceding command should look as follows:

Figure 5.5 – List of Pachyderm Pods

Execute the following command to verify the persistent volumes created as part of the Deployment:

$ kubectl get pv

The output of the preceding command should look as follows:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                         STORAGECLASS         REASON   AGE

pvc-cab1f435-02fb-42df-85d9-d49f6151b281   200Gi      RWO             Delete           Bound    default/etcd-storage-etcd-0   etcd-storage-class            81m

Important note

When Pachyderm is deployed using the --dynamic-etcd-nodes flag, it creates an etcd deployment to manage administrative metadata. Block storage used by etcd Pods is provisioned using the default AWS StorageClass, gp2. To use a different StorageClass during deployment, you will need to deploy an Amazon EBS CSI driver to your cluster and update the etcd.storageclass parameter to gp3 during the Helm Chart deployment.

Execute the following command to verify the successful installation of Pachyderm:

$ pachctl version

The output of the preceding command should look as follows:

COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

Now that we have installed Pachyderm on our AWS EKS cluster, we are ready to create our first pipeline.

Deleting a Pachyderm deployment on Amazon EKS

If you need to delete your Pachyderm deployment or start afresh, you can wipe out your environment and start over again from the Preparing an EKS cluster to run Pachyderm instructions. Let's perform the following steps to delete your existing Pachyderm deployment:

If you have used a different name for your Helm instance, execute the following command to find the Pachyderm instance name deployed using the Helm Chart:

$ helm ls | grep pachyderm

The output of the preceding command should look as follows:

pachd    default         1               2021-12-27 20:20:33.168535538 -0800 PST deployed        pachyderm-2.0.0    2.0.0

Execute the following command using your Pachyderm instance name to remove Pachyderm components from your cluster:

$ helm uninstall pachd

Execute the following command to retrieve the list of EKS clusters deployed and identify the name of your cluster:

$ eksctl get cluster

The output of the preceding command should look as follows:

2021-05-05 21:53:56 [ℹ]  eksctl version 0.47.0

2021-05-05 21:53:56 [ℹ]  using region us-east-1

NAME                            REGION          EKSCTL CREATED

exciting-badger-1620255089      us-east-1       True

If you would like to remove the cluster completely, copy the name of your cluster from the preceding output and execute the following command after replacing the name to delete the Amazon EKS cluster. Note that all the other workloads on this cluster will also be destroyed:

$ eksctl delete cluster --name <name>  

The output of the preceding command should complete similar to the following:

...

2021-05-05 22:00:54 [ℹ]  will delete stack "eksctl-exciting-badger-1620255089-cluster"

2021-05-05 22:00:54 [✔]  all cluster resources were deleted

Now you have completely removed Pachyderm and your EKS cluster from your AWS account.

Deploying Pachyderm on GKE

If you use Google Cloud, a managed Kubernetes service can be deployed on Google Cloud Platform (GCP) using automation and command-line tools with the help of kOps, kubespray, Terraform, and others. For additional configuration details, you can refer to Kubernetes' official documentation at https://kubernetes.io/docs/setup/production-environment/. Let's learn the simplest way to get the services required by Pachyderm up and running on Google Cloud's managed Kubernetes service, GKE.

Preparing a GKE cluster to run Pachyderm

Follow these steps to provision a GKE cluster using the Google Cloud SDK. You will need to have the Google Cloud SDK installed and its credentials configured. If you have a cluster, you can skip these instructions and jump to the Deploying the cluster section. Also, you can refer to the Google Cloud SDK official documentation at https://cloud.google.com/sdk/docs/install:

Execute the following command to deploy a GKE cluster with default parameters. This command will generate a three-node cluster with the recommended n2-standard-4 instance type using a Container-Optimized OS (COS) with Docker in your default compute zone:

$ gcloud container clusters create pachyderm-cluster \

--scopes compute-rw,storage-rw,service-management,service-control,logging-write,monitoring \

--machine-type n2-standard-4

The output of the preceding command should complete similar to the following:

...

kubeconfig entry generated for pachyderm-cluster.

NAME               LOCATION       MASTER_VERSION    MASTER_IP      MACHINE_TYPE   NODE_VERSION      NUM_NODES  STATUS

pachyderm-cluster  us-central1-a  1.18.16-gke.2100  35.238.200.52  n2-standard-4  1.18.16-gke.2100  1          RUNNING

Important note

To simply customize the GKE cluster parameters, you can use the GCP console and the Kubernetes Engine creation wizard. After you configure the parameters on the wizard, click on the command-line button in the wizard to convert the configuration into a CLI command to use with the gcloud command.

Verify deployment of the cluster by executing the following command:

$ kubectl cluster-info && kubectl get nodes

The output of the preceding command should look as follows:

Kubernetes control plane is running at https://<IP_ADDRESS>

GLBCDefaultBackend is running at https://<IP_ADDRESS>/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy

KubeDNS is running at https://<IP_ADDRESS>/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Metrics-server is running at https://<IP_ADDRESS>/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

NAME                                               STATUS    ROLES    AGE     VERSION

gke-pachyderm-cluster-default-pool-26cf3a77-1vr1   Ready    <none>   12      v1.18.16-gke.2100

gke-pachyderm-cluster-default-pool-26cf3a77-5sgs   Ready    <none>   12m     v1.18.16-gke.2100

gke-pachyderm-cluster-default-pool-26cf3a77-lkr4   Ready    <none>   12m     v1.18.16-gke.2100

Now, your GKE cluster is provisioned and ready to deploy Pachyderm.

Creating a Google Cloud object storage bucket

Pachyderm uses object storage to store data. Follow these steps to create a Google Cloud object storage bucket:

Generate variables that will be used to create your Google Cloud Storage (GCS) bucket and pass to pachctl to store your Pachyderm data later. Make sure to use a unique bucket name. In our example, we will use pachyderm-bucket as our bucket name with a capacity of 200 GB and located in the same region as our GKE cluster, us-central1:

$ export GCS_BUCKET_NAME=pachyderm-bucket

$ export GKE_STORAGE_SIZE=200

In order for Pachyderm to store pipeline data, a dedicated GCS bucket is required. Execute the following command to create a GCS bucket with the parameters defined by your variables:

$ gsutil mb gs://${GCS_BUCKET_NAME}

Execute the following command to confirm the creation of the bucket:

$ gsutil ls

The output of the preceding command should look as follows:

gs://pachyderm-bucket/

Now, you have a GCS bucket created to store Pachyderm data. We are ready to deploy Pachyderm on GKE.

Deploying the cluster

When you start learning Pachyderm, it is suggested to run experiments in a small local cluster. We have previously covered the local deployment of Pachyderm in Chapter 4, Installing Pachyderm Locally. In this chapter, we are going to focus on a scalable production-grade deployment of Pachyderm using IAM roles on GKE clusters.

Follow these steps to install Pachyderm on your GKE cluster:

If you don't already have a service account, execute the following command to create a service account:

$ gcloud iam service-accounts create my-service-account --display-name=my-account

Replace the following pachyderm-book in both places with your project name and add a storage admin role that is binding to your service account:

$ gcloud projects add-iam-policy-binding \

pachyderm-book –role roles/owner --member \

serviceAccount:my-service-account@pachyderm-book.iam.gserviceaccount.com    

Now, execute the following command to add Pachyderm Helm Chart repos to your local repository:

$ helm repo add pach https://helm.pachyderm.com  

Execute the following command to get the latest Chart information from the Chart repository:

$ helm repo update

For a quick deployment, replace the Google Cloud bucket name, and Google Cloud credentials, and execute the following command to deploy the latest version of Pachyderm on your cluster without the console:

$ helm install pachd pach/pachyderm \

--set deployTarget=GOOGLE \

--set pachd.storage.google.bucket="GOOGLE_BUCKET" \

--set pachd.storage.google.cred="GOOGLE_CRED" \

--set pachd.externalService.enabled=true

If you have an enterprise key and would like to deploy it with Pachyderm's console user interface, execute the following command:

$ helm install pachd pach/pachyderm \

--set deployTarget=GOOGLE \

--set pachd.storage.google.bucket="GOOGLE_BUCKET" \

--set pachd.storage.google.cred="GOOGLE_CRED" \

--set pachd.enterpriseLicenseKey=$(cat license.txt) \

--set console.enabled=true

Once the console is deployed successfully, follow the instructions under the Accessing the Pachyderm console section to access the console.

A Kubernetes Deployment is a controller that rolls out a ReplicaSet of Pods based on the requirements defined in your manifest file. Execute the following command to verify the state of Deployments created during installation:

$ kubectl get deployments

The output of the preceding command should look as follows:

NAME    READY   UP-TO-DATE   AVAILABLE   AGE

dash    1/1     1            1           44s

pachd   1/1     1            1           45s

Execute the following command to verify a successful installation and see the Pods created as part of the Deployments:

$ kubectl get Pods

The output of the preceding command should look as follows:

NAME                    READY  STATUS   RESTARTS  AGE

dash-cf6f47d7d-xpvvp    2/2    Running  0         104s

etcd-0                  1/1    Running  0         104s

pachd-6c99f6fb7-dnjhn   1/1    Running  0         104s

Execute the following command to verify the persistent volumes created as part of the Deployment:

$ kubectl get pv

The output of the preceding command should look as follows:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                         STORAGECLASS         REASON   AGE

pvc-c4eac147-8571-4ccb-8cd0-7c1cb68a627d 200Gi      RWO            Delete           Bound    default/etcd-storage-etcd-0   etcd-storage-class            3m4s

Execute the following command to verify the successful installation of Pachyderm:

$ pachctl version

The output of the preceding command should look as follows:

COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

Now we have installed Pachyderm on your GKE cluster, you are ready to create your first pipeline.

Deleting a Pachyderm deployment on GKE

If you need to delete your deployment and start afresh, you can wipe out your environment and start over again using the Preparing a GKE cluster to run Pachyderm instructions. Let's perform the following steps to delete your existing Pachyderm deployment:

If you have used a different name for your Helm instance, execute the following command to find the Pachyderm instance name deployed using the Helm Chart:

$ helm ls | grep pachyderm

The output of the preceding command should look as follows:

pachd    default         1               2021-12-27 20:20:33.168535538 -0800 PST deployed        pachyderm-2.0.0    2.0.0

Execute the following command using your Pachyderm instance name to remove the Pachyderm components from your cluster:

$ helm uninstall pachd

Execute the following command to retrieve the list of GKE clusters deployed and identify the name of your cluster:

$ gcloud container clusters list

The output of the preceding command should look as follows:

NAME               LOCATION       MASTER_VERSION    MASTER_IP      MACHINE_TYPE   NODE_VERSION      NUM_NODES  STATUS

pachyderm-cluster  us-central1-a  1.18.16-gke.2100  35.238.200.52  n2-standard-2  1.18.16-gke.2100  3          RUNNING

If you would like to remove the cluster completely, copy the name of your cluster from the preceding output and execute the following command after replacing the name to delete the GKE cluster. Note that all the other workloads on this cluster will also be destroyed:

$ gcloud container clusters delete <name>

The output of the preceding command should complete similar to the following:

...

Deleting cluster pachyderm-cluster...done.

Deleted [https://container.googleapis.com/v1/projects/pachydermbook/zones/us-central1-a/clusters/pachyderm-cluster].

Now you have completely removed Pachyderm and your Kubernetes cluster from your GCP account.

Deploying Pachyderm on Microsoft AKS

If you use Microsoft Azure, a managed Kubernetes service can be deployed on the Azure platform using automation and command-line tools with the help of kOps, kubespray, Terraform, and others. For additional configuration details, you can refer to Kubernetes' official documentation at https://kubernetes.io/docs/setup/production-environment/. Let's learn the simplest way to get the services required by Pachyderm up and running on AKS.

Preparing an AKS cluster to run Pachyderm

Follow these steps to provision an AKS cluster using the Azure CLI. You will need to have the Azure CLI installed and its credentials configured. If you have a cluster, you can skip these instructions and jump to the Deploying Pachyderm on Microsoft AKS section. Also, you can refer to the Azure CLI official documentation at https://docs.microsoft.com/en-us/cli/azure/:

Execute the following command in your previously specified resource-group to deploy an AKS cluster with default parameters. This command will generate a three-node cluster with the recommended Standard_DS4_v2 instance type in your default compute zone:

$ az aks create --resource-group pachyderm-group --name pachyderm-cluster --generate-ssh-keys --node-vm-size Standard_DS4_v2

The output of the preceding command should complete similar to the following:

...

   "privateFqdn": null,

  "provisioningState": "Succeeded",

  "resourceGroup": "pachyderm-group",

  "servicePrincipalProfile": {

    "clientId": "msi",

    "secret": null

  },...

Important note

If you don't remember your resource group name, you can use the az group list command to list the previously created resource groups.

Execute the following command to connect to your cluster:

$ az aks get-credentials --resource-group pachyderm-group --name pachyderm-cluster

Verify the cluster deployment by executing the following command:

$ kubectl get nodes

The output of the preceding command should look as follows:

NAME                                STATUS   ROLES   AGE     VERSION

aks-nodepool1-34139531-vmss000000   Ready    agent   5m57s   v1.19.9

aks-nodepool1-34139531-vmss000001   Ready    agent   5m58s   v1.19.9

aks-nodepool1-34139531-vmss000002   Ready    agent   5m58s   v1.19.9

Now your AKS cluster is provisioned and ready to deploy Pachyderm.

Creating an Azure storage container

Pachyderm uses blob storage to store data and block storage for metadata. It is recommended to use SSDs rather than the Standard HDD-based slower storage option.

Follow these steps to create a Premium LRS Block blobs storage container:

Generate variables that will be used to create your Azure Block blobs storage and pass to pachctl to store your Pachyderm data later. Make sure to use a unique storage account and container name. In our example, we will use pachydermstorageaccount as our STORAGE_ACCOUNT, pachydermblobcontainer as our CONTAINER_NAME, and it will be located in the centralus region:

$ export RESOURCE_GROUP=pachyderm-group

$ export STORAGE_ACCOUNT=pachydermstorageaccount

$ export CONTAINER_NAME=pachydermblobcontainer

$ export AZURE_STORAGE_SIZE=200

Execute the following command to your Azure storage account with the parameters defined by your variables:

$ az storage account create \

  --resource-group="${RESOURCE_GROUP}" \

  --location="centralus" \

  --sku=Premium_LRS \

  --name="${STORAGE_ACCOUNT}" \

  --kind=BlockBlobStorage

Execute the following command to confirm the creation of the bucket:

$ az storage account list

The output of the preceding command should look as follows:

...

    "web": "https://pachydermstorageaccount.z19.web.core.windows.net/"

  },

  "primaryLocation": "centralus",

  "privateEndpointConnections": [],

  "provisioningState": "Succeeded",

  "resourceGroup": "pachyderm-group",

...

The deployment of Pachyderm will require a storage account key. Execute the following command to store the storage keys in a variable:

$ STORAGE_KEY="$(az storage account keys list \

              --account-name="${STORAGE_ACCOUNT}" \

              --resource-group="${RESOURCE_GROUP}" \

              --output=json \

              | jq '.[0].value' -r

            )"

Execute the following command to create a data storage container in your storage account:

$ az storage container create --name ${CONTAINER_NAME} \

          --account-name ${STORAGE_ACCOUNT} \

          --account-key "${STORAGE_KEY}"

The output of the preceding command should look as follows:

{

  "created": true

}

Now, you have an Azure data storage container created in your Azure storage account to store Pachyderm data.

Deploying the cluster

When you start learning Pachyderm, it is suggested to run experiments in a small local cluster. We have previously covered the local deployment of Pachyderm in Chapter 4, Installing Pachyderm Locally. In this chapter, we are going to focus on a scalable production-grade deployment of Pachyderm on an AKS cluster.  

Follow these steps to install Pachyderm on your AKS cluster:

If you have not connected to your Kubernetes cluster, execute the following command to connect to your cluster:

$ az aks get-credentials --resource-group pachyderm-group --name pachyderm-cluster

For a quick deployment, replace the Azure storage container name, Azure storage account name, and AKS account key, and execute the following command to deploy the latest version of Pachyderm on your cluster without the console:

$ helm install pachd pach/pachyderm \

--set deployTarget=MICROSOFT \

--set pachd.storage.microsoft.container="CONTAINER_NAME" \

--set pachd.storage.microsoft.id="AZURE_ID" \

--set pachd.storage.microsoft.secret="AZURE_SECRET" \

--set pachd.externalService.enabled=true

If you have an enterprise key and you would like to deploy it with Pachyderm's console user interface, execute the following command:

$ helm install pachd pach/pachyderm \

--set deployTarget=MICROSOFT \

--set pachd.storage.microsoft.container="CONTAINER_NAME" \

--set pachd.storage.microsoft.id="AZURE_ID" \

--set pachd.storage.microsoft.secret="AZURE_SECRET" \

--set pachd.enterpriseLicenseKey=$(cat license.txt) \

--set console.enabled=true

Once the console is deployed successfully, follow the instructions under the Accessing the Pachyderm console section to access the console.

A Kubernetes Deployment is a controller that rolls out a ReplicaSet of Pods based on the requirements defined in your manifest file. Execute the following command to verify the state of Deployments created during the installation:

$ kubectl get deployments

The output of the preceding command should look as follows:

NAME    READY   UP-TO-DATE   AVAILABLE   AGE

dash    1/1     1            1           39s

pachd   1/1     1            1           39s

Execute the following command to verify a successful installation and see the Pods created as part of the Deployments:

$ kubectl get pods

The output of the preceding command should look as follows:

NAME                    READY   STATUS    RESTARTS   AGE

dash-866fd997-z79jj     2/2     Running   0          54s

etcd-0                  1/1     Running   0          54s

pachd-8588c44f56-skmkl  1/1     Running   0          54 s

Execute the following command to verify the persistent volumes created as part of the Deployment:

$ kubectl get pv

The output of the preceding command should look as follows:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                         STORAGECLASS   REASON   AGE

pvc-9985a602-789d-40f3-9249-7445a9c15bc3   200Gi      RWO            Delete           Bound    default/etcd-storage-etcd-0   default                 89s

Important note

When Pachyderm is deployed using the --dynamic-etcd-nodes flag, it creates an etcd deployment to manage administrative metadata. In Azure, block storage used by etcd Pods is provisioned using the default StorageClass. This Storage Class uses the azure-disk provisioner with StandardSSD_LRS volumes. To use a different StorageClass during the deployment, you can customize the values.yaml file and update the etcd.storageClass parameter prior to the deployment.

Execute the following command to verify the successful installation of Pachyderm:

$ pachctl version

The output of the preceding command should look as follows:

COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

Now that we have installed Pachyderm on your AKS cluster, you are ready to create your first pipeline.

Deleting a Pachyderm deployment on AKS

If you need to delete your deployment or start afresh, you can wipe out your environment and start over again using the Preparing an AKS cluster to run Pachyderm instructions.  

Let's perform the following steps to delete your existing Pachyderm deployment:

If you have used a different name for your Helm instance, execute the following command to find the Pachyderm instance name deployed using the Helm Chart:

$ helm ls | grep pachyderm

The output of the preceding command should look as follows:

pachd    default         1               2021-12-27 20:20:33.168535538 -0800 PST deployed        pachyderm-2.0.0    2.0.0

Execute the following command using your Pachyderm instance name to remove Pachyderm components from your cluster:

$ helm uninstall pachd

Execute the following command to retrieve the list of AKS clusters deployed and identify the name of your cluster:

$ az aks list

The output of the preceding command should look as follows:

...

    "location": "centralus",

    "maxAgentPools": 100,

    "name": "pachyderm-cluster",

    "networkProfile": {

…

If you would like to remove the cluster completely, copy the name of your cluster from the preceding output and execute the following command after replacing the name to delete the AKS cluster. Note that all the other workloads on this cluster will also be destroyed:

$ az aks delete --name <name> --resource-group pachyderm-group

The output of the preceding command should complete similar to the following:

...

Deleting cluster pachyderm-cluster...done.

Deleted [https://container.googleapis.com/v1/projects/pachydermbook/zones/us-central1-a/clusters/pachyderm-cluster].

Now you have completely removed Pachyderm and your Kubernetes cluster from your AKS account.

Accessing the Pachyderm console

Pachyderm Enterprise Edition offers a graphical user interface where you can see pipelines and repositories. Accessing the Pachyderm console using port forwarding was covered in Chapter 4, Installing Pachyderm Locally.

In addition, for cloud deployments, you can deploy a Kubernetes ingress to access the Pachyderm console securely. For more information, refer to the official Pachyderm documentation.

Summary

In this chapter, we learned the software prerequisites for getting Pachyderm up and running on managed Kubernetes services from major cloud providers including AWS, Google Cloud, and Microsoft Azure.

We acquired basic knowledge of cloud providers' command-line tools and learned how to install and operate them on your local machine to provide production-grade Kubernetes clusters.

We created an object storage bucket and also deployed highly available multi-node managed Kubernetes clusters using the most common configuration options. And finally, we deployed a Pachyderm instance.

In the next chapter, we will learn in detail about creating your first pipeline. You will learn a simple data science example and a pipeline creation workflow.

Further reading

You can refer to the following links for more information on the topics covered in this chapter:

Using the AWS CLI official documentation: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-using.html

Getting started with the Google Cloud SDK official documentation: https://cloud.google.com/sdk/docs/quickstart

Using the Azure CLI official documentation: https://docs.microsoft.com/en-us/cli/azure/

eksctl documentation: https://eksctl.io/introduction/

eksctl advanced cluster deployment examples: https://github.com/weaveworks/eksctl/tree/main/examples

kOps documentation: https://kubernetes.io/docs/setup/production-environment/tools/kops/

Kubespray documentation: https://kubernetes.io/docs/setup/production-environment/tools/kubespray/

Kubernetes in production best practices: https://www.packtpub.com/product/kubernetes-in-production-best-practices/9781800202450

Mastering Kubernetes – Third Edition: https://www.packtpub.com/product/mastering-kubernetes-third-edition/9781839211256

Chapter 6: Creating Your First Pipeline

In Chapter 3, Pachyderm Pipeline Specification, we learned about the Pachyderm pipeline specification and what parameters you can configure in it. The pipeline specification is the most critical configuration piece of your pipeline, along with your code. In this chapter, we will learn how to create a Pachyderm pipeline that performs image processing. We will walk through all the steps that are involved in this process, including creating the Pachyderm repository, creating a pipeline, viewing the results of our computations, and adding an extra step to our original pipeline.

In this chapter, we will cover the following topics:

Pipeline overview

Creating a repository

Creating a pipeline specification

Viewing the pipeline result

Adding another pipeline step

Technical requirements

This chapter requires that you have access to the following components and that they are installed and configured.

For a local macOS installation, you will need the following:

macOS Mojave, Catalina, Big Sur, or later

Docker Desktop for Mac 10.14

minikube v1.19.0 or later

pachctl 2.0.x or later

Pachyderm 2.0.x or later

For a local Windows installation, you will need the following:

Windows Pro 64-bit v10 or later

Windows Subsystem for Linux (WSL) 2 or later

Microsoft Powershell v6.2.1 or later

Hyper-V

minikube v1.19.0 or later

kubectl v1.18 or later

pachctl 2.0.x or later

Pachyderm 2.0.x or later

For an Amazon Elastic Kubernetes Service (Amazon EKS) installation, you will need the following:

kubectl v.18 or later

eksctl

aws-iam-authenticator

pachctl 2.0.x or later

Pachyderm 2.0.x or later

For a Microsoft Azure cloud installation, you will need the following:

kubectl v.18 or later

The Azure CLI

pachctl 2.0.x or later

Pachyderm 2.0.x or later

jq 1.5 or later

For a Google Kubernetes Engine (GKE) cloud installation, you will need the following:

Google Cloud SDK v124.0.0. or later

kubectl v.18 or later

pachctl 2.0.x or later

Pachyderm 2.0.x or later

You do not need any special hardware to be able to run the pipelines in this chapter. If you are running your Pachyderm cluster locally, any modern laptop should support all the operations in this chapter. If you are running Pachyderm in a cloud platform, you will need to have a Persistent Volume (PV). See Chapter 5, Installing Pachyderm on a Cloud Platform, for more details.

All the scripts and data described in this chapter are available at https://github.com/PacktPublishing/Reproducible-Data-Science-with-Pachyderm/tree/main/Chapter06-Creating-Your-First-Pipeline.

Now that we have reviewed the technical requirements for this chapter, let's take a closer look at our pipeline.

Pipeline overview

In Chapter 4, Installing Pachyderm Locally, and Chapter 5, Installing Pachyderm on a Cloud Platform, we learned how to deploy Pachyderm locally or on a cloud platform. By now, you should have some version of Pachyderm up and running, either on your computer or a cloud platform. Now, let's create our first pipeline.

A Pachyderm pipeline is a technique that processes data from a Pachyderm input repository or repositories and uploads it to a Pachyderm output repository. Every time new data is uploaded to the input repository, the pipeline automatically processes it. Every time new data lands in the repository, it is recorded as a commit hash and can be accessed, rerun, or analyzed later. Therefore, a pipeline is an essential component of the Pachyderm ecosystem that ensures the reproducibility of your data science workloads.

To get you started quickly, we have prepared a simple example of image processing that will draw a contour on an image. A contour is an outline that represents the shape of an object. This is a useful technique that is often applied to image processing pipelines.

Image processing is a widely used technique that enables you to enhance the quality of images, transform an image into another image, extract various information about an image, and so on. With machine learning, you can set up pipelines that can determine objects in an image, create a histogram of an image, and so on.

There are many open source libraries that you can use for advanced image processing, with the most famous among them being OpenCV and scikit-image. Both of these libraries are widely used by machine learning experts for various image processing tasks.

For this example, we will use scikit-image. Scikit-image, or skimage, is an open-source image processing library that enables you to run various image processing algorithms to analyze and transform images. Scikit-image is built to work with the Python programming language.

In this example, we will use scikit-image with a couple of other open source components, including the following:

NumPy: An open source Python library that enables you to work with arrays. When you need to analyze or segment an image, it must be transformed into an array for processing.

Matplotlib: An extension to NumPy that enables you to plot images and create data visualizations.

We will create a pipeline that will consume the data from the photos repository, run a contour.py script against the images in the photos repository, and upload the result to the contour output repository.

The following diagram explains our pipeline:

Figure 6.1 – Contour pipeline

The following code explains the contour.py script:

import numpy as np

import matplotlib.pyplot as plt

import os

from skimage.io import imread, imsave

from skimage import measure

import logging

logging.basicConfig(level=logging.INFO)

def create_contours(photo):

     image = imread(photo, as_gray=True)

     t = os.path.split(photo)[1]

     c = measure.find_contours(image, 0.9)

     fig, ax = plt.subplots()

     ax.imshow(image, vmin=-1, vmax=1, cmap='gray')

     for contour in c:

         ax.plot(contour[:, 1], contour[:, 0], linewidth=2)

     ax.axis('image')

     ax.set_xticks([])

     ax.set_yticks([])

     plt.savefig(os.path.join("/pfs/out", os.path.splitext(t)[0]+'.png'))

for dirpath, dirnames, filenames in os.walk("/pfs/photos"):

     for f in filenames:

         create_contours(os.path.join(dirpath, f))

The preceding script contains a function called create_contours. This function does the following:

First, it reads the image file from the pfs/photos repository and converts it into a grayscale image. This is needed to transform a color image (RGB) into a two-dimensional NumPy array.

Then, it uses the measure.find_contours API method from the skimage.measure.find_contours module to find the contour of the two-dimensional array that we converted our image into at a value of 0.9. This value represents the position between light and dark tones. Typically, it's best to use a middle value, but in this case, 0.9 created the best results.

Then, it defined subplots to visualize our image and saved it in the pfs/out directory, which, in reality, will be the pfs/out/contour output repository.

The last part of the script tells the program to apply the create_contours function to all the files in the pfs/photos repository.

This script is built into a Docker image that we will use to run our pipeline. This Docker image is hosted in Docker Hub.

We will use the following images from freepik.com for processing. The first image is an image of a brown vase:

Figure 6.2 – Brown vase

The second image is an image of a hand:

Figure 6.3 – Hand

Finally, the third image is an image of a landscape:

Figure 6.4 – Landscape

As you can see, these are some simple images where finding the contours should be easy. You can try to run this pipeline against more complex images and see the results you'll get. In general, the more contrast you have between the elements in the image, the more precise contours the algorithm will be able to find.

Now that we understand the example we are working on, let's go ahead and create a repository.

Creating a repository

The first step in creating your pipeline is to create a Pachyderm repository and put some data in it. As you probably remember from Chapter 2, Pachyderm Basics, a Pachyderm repository is a location inside of a Pachyderm cluster where you store your data. We will be creating an input repository, and the pipeline will automatically create an output repository on the first run.

To create an input repository, perform the following steps:

Log into your Terminal.

Verify that Pachyderm is up and running:

% pachctl version

You should see an output similar to the following:

COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

The pachd component must list a version.

Create a Pachyderm input repository called photos:

% pachctl create repo photos

No output will be returned.

Verify that the photos repository was created:

% pachctl list repo

You should see the following output:

NAME   CREATED        SIZE (MASTER) ACCESS LEVEL

photos 11 seconds ago ≤ 0B          [repoOwner]

Here, you can see that the photos repository was created and that it is empty. Although the size is counted for the master branch, at this point, no branches exist in this repository. Pachyderm will automatically create a specified branch when you put files in it.

Put some images in the photos repository. We need to put the files into the root of our photos repository. To do this, you need to use the -r (recursive) flag and specify the path to the directory that contains the files on your computer. For example, if you have downloaded the files to a data folder on your computer, you need to run the following command:

% pachctl put file -r photos@master:/ -f data

The following is some sample output:

data/brown_vase.png 25.82 KB / 25.82 KB [=====] 0s 0.00 b/s

data/hand.png 33.21 KB / 33.21 KB [======] 0s 0.00 b/s

data/landscape.png 54.01 KB / 54.01 KB [=========] 0s

Pachyderm automatically creates the specified branch. In this example, Pachyderm creates the master branch. You could name the branch anything you want, but for simplicity, let's call it master. All the commands in this section are written with the master branch in mind.

Verify that the files were added to the repository:

% pachctl list file photos@master

The following is some sample output:

NAME           TYPE SIZE

/brown_vase.png file 25.21KiB

/hand.png       file 32.43KiB

/landscape.png  file 52.74KiB

Alternatively, you could place each file one by one by running the pachctl put file command for each image. For example, to place the landscape.jpg file, change the directory on your computer to data and use the following command:

% pachctl put file photos@master -f landscape.jpg

The following is some sample output:

data/landscape.jpg 54.01KB / 54.01 KB [=] 0s 0.00 b/s/

Repeat this command for all image files.

Important note

Make sure that the TYPE parameter in the output says file and not dir. Pachyderm does not distinguish between directories and files, and you use -f to put files in a directory or the root of the repository. If any of your files are listed as dir, you need to delete them by running pachctl delete file photos@master:<path> and start over. The pipeline will not work as expected if you place the files in the directories.

Now that we have created a repository, let's create our pipeline specification.

Creating a pipeline specification

In the Creating a repository section, we created a repository called photos and put some test files in it. The pipeline specification for this example must reference the following elements:

An input repository with data to process

A computer program or a script that needs to run against your data

A glob pattern that specifies the datum granularity

A Docker image with built-in dependencies that contains your code

We have created a pipeline specification for you so that you can use it to create the pipeline. Here is what is in the pipeline specification file in the YAML Ain't Markup Language (YAML) format:

---

pipeline:

   name: contour

description: A pipeline that identifies contours on an image.

transform:

   cmd:

   - python3

   - "/contour.py"

   image: svekars/contour-histogram:1.0

input:

   pfs:

     repo: photos

     glob: "/"

Let's look at the pipeline specification in more detail. Here are the parameters in the pipeline:

Figure 6.5 – Contour pipeline parameters

This pipeline specification does not include any optimization or other extra parameters. It is a minimum pipeline that will do the required computation for our example.

To create a pipeline, perform the following steps:

Log into your terminal.

Create the contour pipeline by running the following command:

% pachctl create pipeline -f contour.yaml

No output will be returned.

Verify that the pipeline has been created:

% pachctl list pipeline

The following is the system output:

NAME    VERSION INPUT    CREATED       STATE / LAST JOB   DESC

contour 1       photos:/* 5 seconds ago running / - A pipeline that identifies contours on an image.

As soon as you create the pipeline, it will set its status to running and attempt to process the data in the input repository. You might also see that LAST JOB is set to starting or running.

Important note

Another important thing in the preceding output is the pipeline version. The version of our pipeline is 1. If you change anything in your pipeline's YAML file and update your pipeline after that, the version counter will be updated to a subsequent number.

View the job that has been started for your pipeline:

pachctl list job

The following is the system output:

ID       SUBJOBS PROGRESS CREATED       MODIFIED

71169d…  1                      2 minutes ago 2 minutes ago

The output of the pachctl list job command gives us the following information:

Figure 6.6 – Pipeline output explained

Now that our pipeline has run successfully, let's see the output in the output repository.

Viewing the pipeline result

Once your pipeline has finished running, you can view the result in the output repository. We will look at the output result in both the command line and the Pachyderm dashboard for visibility.

If you are using a local Pachyderm deployment with minikube, you need to enable port-forwarding before you can access the Pachyderm UI.

To view the pipeline result in the terminal, perform the following steps:

Log into your terminal.

Verify that the output repository called contour has been created:

% pachctl list repo

The following is the system output:

% pachctl list repo

NAME    CREATED            SIZE (MASTER) ACCESS LEVEL

contour About a minute ago ≤ 117.6KiB    [repoOwner]  Output repo for pipeline contour.

photos  5 minutes ago      ≤ 110.4KiB    [repoOwner] B

As you can see, the contour repository has been created, and it contains 117.6 KiB of data. If you are running Pachyderm locally, you can also preview what those files look like by running one of the following commands.

If you are on macOS, run the following command to list the files in the output repository:

% pachctl list file contour@master

The following is the system output:

NAME             TYPE SIZE

/brown_vase.png file 23.78KiB

/hand.png       file 29.79KiB

/landscape.png  file 64.08Kib

Now, pipe the open command to the pachctl get file command to open one of the images in the default preview application on your Mac. For example, to preview hand.png, run the following command:

% pachctl get file contour@master:hand.png | open -f -a Preview.app

You should see the following output:

Figure 6.7 – Hand processed

Now, let's view the processed results in the UI. If you are running Pachyderm on a cloud provider, just point your browser to the IP address where the Pachyderm dashboard is running. If you are running Pachyderm in Pachyderm Hub, follow the instructions in Pachyderm Hub to access the console. If you are running Pachyderm locally in minikube, follow the remaining steps to enable port-forwarding.

Important note

Unless you are running your experiments in Pachyderm Hub, the Pachyderm console is only available if you have deployed it. You need to have a trial or an enterprise license to deploy the Pachyderm console locally or in the cloud.

Open a separate terminal window.

Enable port forwarding for your local Pachyderm deployment by running the following command:

% pachctl port-forward

You should see the following output:

Forwarding the pachd (Pachyderm daemon) port...

listening on port 30650

Forwarding the pachd (Pachyderm daemon) port...

listening on port 30650

Forwarding the OIDC callback port...

listening on port 30657

Forwarding the s3gateway port...

listening on port 30600

Forwarding the identity service port...

listening on port 30658

Forwarding the dash service port...

...

If you have used the default settings, the Pachyderm dashboard must load at http://localhost:30080.

Paste the dashboard IP address into a web browser.

If you are prompted to log in, follow the onscreen instructions to do so. When you log in, you should see the following screen:

Figure 6.8 – Pachyderm Direct Acyclic Graph (DAG)

This is a Direct Acyclic Graph (DAG) that's been created for the input and output repositories, as well as the pipeline.

Click on the contour output repository (the last one on the screen) and then click View Files:

Figure 6.9 – Output repository information

A list of files will appear.

Click Preview next to hand.png:

Figure 6.10 – Output file

You can preview all the resulting files from the UI in the same way. From here, the files can be either consumed by another pipeline or served outside of Pachyderm through an S3 Gateway, or the output repository can be mounted and accessible on a local disk. Let's look at what the other images should look like.

You should see that the landscape image has changed, as follows:

Figure 6.11 – Processed landscape image

The brown_vase.png image should look like this:

Figure 6.12 – Processed vase image

In this section, we learned how to view the results of a pipeline. Now, let's add another pipeline step.

Adding another pipeline step

Pachyderm pipelines can be chained into multi-step workflows. For each step, you will need to have a separate pipeline specification and a Docker image if you are using one. In this section, we will add another step to our image processing workflow. We will use the skimage.exposure.histogram module to create histograms of all the images we have in our contour output repository.

Example overview

A histogram is a visual representation of data that provides information about the image, such as the number of pixels, their intensity, and other information. Because we represent images as numerical data, we can create a histogram for each of the images we processed in the first step of our workflow – the contour pipeline. In this new step of the workflow, we will create histograms for each image that has landed in the contour output repository and save them in the histogram output repository in PNG format.

Here is an example of a histogram that has been generated for the hand.png image:

Figure 6.13 – Grayscale image histogram

The y-axis represents the count of pixels, while the x-axis represents the intensity of the pixels.

Here is a diagram of the new two-step workflow, which includes the contour and histogram pipelines:

Figure 6.14 – Contour and histogram workflow

The histogram pipeline will consume the files from the contour repository, create histograms for them, and output them to the histogram repository.

Histogram creation script overview

In this pipeline step, we will use histogram.yaml, which will create a pipeline for us. The pipeline will run the histogram.py script.

Let' review the histogram.py script, which creates histograms from the file in the contour repository:

from skimage import io

import matplotlib.pyplot as plt

import numpy as np

import os

def create_histogram(photo):

    image = io.imread(photo)

    t = os.path.split(photo)[1]

    plt.hist(image.ravel(), bins=np.arange(10, 70),  color='blue', alpha=0.5, rwidth=0.7)

    plt.yscale('log')

    plt.margins(x=0.03, y=-0.05)

    plt.xlabel('Intensity')

    plt.ylabel('Count')

    plt.savefig(os.path.join("/pfs/out", os.path.splitext(t)[0]+'.png'))

for dirpath, dirnames, filenames in os.walk("/pfs/contour"):

    for f in filenames:

        create_histogram(os.path.join(dirpath, f))

This script imports the following libraries:

skimage.io: The io module from the scikit-image library enables read and write operations within your Python file. We need this module to read files from the contour repository.

matploit.pyplot: This Matplotlib interface enables us to plot our images. We use it to create the histogram, add labels to the x- and y-axis of the plot, and so on.

numpy: We need NumPy to represent the image as an array and keep the number of bins in the required range.

os: The os module from the standard Python library enables read and write operations with files. We need this module to read images from the Pachyderm contour repository and save our images in the correct output repository.

Let's take a closer look at what the script does. The create_histogram function reads image files from the contour repository. Then, using the matploit.pyplot.hist (plt.hist) function, the script creates a histogram with the following parameters:

The numpy.ravel function converts our images from the 2D array into a 1D array, which is needed to plot a histogram.

The bins parameter defines the shape and distribution of vertical bars in your histogram. To distribute them evenly on the plot, we have a defined range by using the np.arange function.

The color='blue' parameter defines the color of the histogram bins.

The alpha=0.5 parameter defines the level of transparency.

The rwidth=0.7 parameter defines the width of each bin.

The plt.yscale('log') parameter defines the logarithmic y-axis scale. We need this parameter to narrow the scale of the y-axis for better data visualization.

The plt.margins(x=0.03, y=-0.05) parameter determines the amount of whitespace between the histogram and the start of the plot.

The plt.xlabel('Intensity') and plt.ylabel('Count') parameters define the labels for the x- and y-axes.

The plt.savefig function defines where to save the histogram. In our case, we will save it in the pfs/out directory, in which Pachyderm will automatically create a histogram repository under the pfs/out/histogram path.

Important note

In your scripts, you do not need to add the path to the repository, just to the pfs/out directory.

Pipeline specification overview

The histogram.yaml pipeline specification creates the histogram pipeline.

Here is what our pipeline specification will look like:

---

pipeline:

   name: histogram

description: A pipeline that creates histograms for images stored in the contour repository.

transform:

   cmd:

   - python3

   - "/histogram.py"

   image: svekars/contour-histogram:1.0

input:

   pfs:

     repo: contour

     glob: "/"

This pipeline does the following:

Uploads the files stored in the contour repository as a single datum

Pulls the Docker image stored in Docker Hub under svekars/histogram:1.0

Runs the histogram.py script against all the files that have been downloaded from the contour repository

Uploads the results of the transformation to the histogram output repository

Now that we have reviewed what goes into this pipeline, let's go ahead and create it.

Creating the pipeline

The next step is to create the histogram pipeline, which will create a histogram for each image in the photos repository.

Let's create the second step of our workflow by using the histogram.yaml file:

Log into your terminal.

Verify that Pachyderm is up and running:

% pachctl version

The following is the system output:

COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

If pachd is unresponsive, you might need to run minikube stop and minikube start in a local installation to be able to resume working with it. If you are in a cloud environment, you will need to check your connection. If you are running Pachyderm in Pachyderm Hub, check that you are authenticated from the console and follow the onscreen instructions.

Create the histogram pipeline:

% pachctl create pipeline -f histogram.yaml

No system output will be returned.

Verify that the pipeline was created:

% pachctl list pipeline

The following is the system output:

NAME      VERSION INPUT     CREATED        STATE / LAST JOB  DESCRIPTION

contour   1       photos:/* 12 minutes ago running / success A pipeline that identifies contours on an image.

histogram 1       contour:/ 12 seconds ago running / success A pipeline that creates histograms for images stored in the contour repository.

According to the preceding output, the histogram pipeline was created and is currently running the code.

Get the list of repositories:

% pachctl list repo

The following is the system output:

NAME      CREATED            SIZE (MASTER) DESCRIPTION

histogram 23 seconds ago ≤ 27.48KiB    [repoOwner]  Output repo for pipeline histogram.

contour   24 minutes ago ≤ 117.6KiB    [repoOwner]  Output repo for pipeline contour.

photos    29 minutes ago ≤ 110.4KiB    [repoOwner]   

The histogram output repository was created and contains 27.48 KiB of data. These are our histogram files.

List the files in the histogram repository:

% pachctl list file histogram@master

The following is the system output:

NAME           TYPE     SIZE

/hand.png      file 9.361KiB

/landscape.png file 9.588KiB

/brown_vase.png  file 8.526KiB

With that, our histogram visualizations have been added to the repository.

View the histogram files. For example, if you are on Mac, to view the landscape.png histogram file, run the following command:

pachctl get file histogram@master:landscape.png | open -f -a Preview.app

Here is the resulting histogram:

Figure 6.15 – Histogram of the landscape image

You can preview other files in a similar way or through the Pachyderm dashboard, as described earlier.

Go to the console and view the DAG for this newly added pipeline:

Figure 6.16 – Updated DAG

As you can see, you now have a new pipeline called histogram and a new eponymous output repository added to your DAG.

Now that we have created our first pipeline, let's clean up our environment so that we have a clean cluster to work on the tasks in the next chapter.

Cleaning up

Once you are done experimenting, you might want to clean up your cluster so that you start your next experiment with a fresh install. To clean up the environment, perform the following steps:

Delete all the pipelines and repositories:

pachctl delete pipeline –all && pachctl delete repo --all

Verify that no repositories and pipelines exist in your cluster:

pachctl list repo && pachctl list pipeline

You should see the following output:

NAME CREATED SIZE (MASTER) DESCRIPTION

NAME VERSION INPUT CREATED STATE / LAST JOB DESCRIPTION

With that, you have successfully cleaned up your cluster.

Summary

In this chapter, you successfully created your first Pachyderm repository, pipeline, and even extended it with another pipeline step. We used scikit-image, Matplotlib, and NumPy to create contours on images stored in Pachyderm repositories and created histograms for all of these images. This is the first step in understanding how Pachyderm works. In Pachyderm, you'll work with pipelines quite a lot. As you have already noticed, you can put any code in your pipeline. Although most of the examples in this book will use Python, you can use any programming language of your choice.

In the next chapter, we will learn more about Pachyderm functionality, how to ingest data into Pachyderm and export it to outside systems, how to make changes to a pipeline's code, how to tune various parameters, and other important Pachyderm operations.

Further reading

For more information about the topics that were covered in this chapter, take a look at the following resources:

Docker Hub documentation: https://docs.docker.com/docker-hub/

Matplotlib documentation: https://matplotlib.org/

NumPy documentation: https://numpy.org/doc/stable/

Scikit-image documentation: https://scikit-image.org

Landscape image: https://www.freepik.com/free-vector/beautiful-gradient-spring-landscape_6969720.htm

Brown vase image: https://www.freepik.com/free-photo/narrow-neck-vase_921733.htm

Hand image: https://www.freepik.com/free-photo/hand-holding-something-with-white-background_978615.htm

Chapter 7: Pachyderm Operations

In Chapter 6, Creating Your First Pipeline, we created our first pipeline, as well as learning how to create Pachyderm repositories, put data into a repository, create and run a pipeline, and view the results of the pipeline. We now know how to create a standard Pachyderm pipeline specification and include our scripts in it so that they can run against data in our input repository.

In this chapter, we will review all the different ways to put data inside of Pachyderm and export it to outside systems. We will learn how to update the code that runs inside of your pipeline and what the process of updating the pipeline specification is. We will learn how to build a Docker container and test it locally before uploading it to a registry.

We will also look into the most common troubleshooting steps that you should perform when a pipeline fails.

This chapter will cover the following topics:

Reviewing the standard Pachyderm workflow

Executing data operations

Executing pipeline operations

Running maintenance operations

Technical requirements

You should have already installed the following components.

For a local macOS installation, you need the following:

macOS Mojave, Catalina, Big Sur, or later

Docker Desktop for Mac 10.14

minikube v1.9.0 or later

pachctl 2.0.x or later

Pachyderm 2.0.x or later

For a local Windows installation, you need the following:

Windows Pro 64-bit v10 or later

Windows Subsystem for Linux (WSL) 2 or later

Microsoft PowerShell v6.2.1 or later

Hyper-V

minikube v1.9.0 or later

kubectl v1.18 or later

pachctl 2.0.x or later

Pachyderm 2.0.x or later

For an Amazon Elastic Kubernetes Service (Amazon EKS) installation, you need the following:

kubectl v.18 or later

eksctl

aws-iam-authenticator

pachctl 2.0.x or later

Pachyderm 2.0.x or later

For a Microsoft Azure Kubernetes Service (AKS) cloud installation, you need the following:

kubectl v.18 or later

Azure CLI

pachctl 2.0.x or later

Pachyderm 2.0.x or later

jq 1.5 or later

For a Google Kubernetes Engine (GKE) cloud installation, you need the following:

Google Cloud SDK v124.0.0. or later

kubectl v.18 or later

pachctl 2.0.x or later

Pachyderm 2.0.x or later

In addition to this, you need to have the following:

A Docker Hub account to be able to upload images

Downloading the source files

All the source files for this chapter are located in this repository: https://github.com/PacktPublishing/Reproducible-Data-Science-with-Pachyderm/tree/main/Chapter07-Pachyderm-Operations.

Reviewing the standard Pachyderm workflow

As you probably noticed when you were creating a pipeline, there is a certain workflow that you will need to follow when working with Pachyderm. Depending on your automation tools, your team processes, and the software that you use, it might differ, but in general, it boils down to the following common steps:

Add your data to Pachyderm.

Create a pipeline or pipelines.

Add more data.

Modify the parameters.

Modify your pipeline or pipelines.

Output the result to an output repository.

Repeat Steps 3–6 as many times as needed or continuously.

Serve your model or package your library and make it available for use.

The following diagram demonstrates this process:

Figure 7.1 – Pachyderm workflow

Depending on whether you keep your code in a Docker image, in the pipeline itself, or you use a build pipeline with your Python code, you need to rebuild your Docker image every time you make changes to the code. There is a lightweight workflow for Python pipelines only that uses a base Docker image and a special build pipeline. You can read about this approach in the Pachyderm official documentation at https://docs.pachyderm.com. For any other language, you likely will need to build Docker images regularly.

Now that we know what the typical workflow is, let's dive into data operations and learn about all the ways you can upload your data to Pachyderm.

Executing data operations

As you have probably already noticed, to get started working with Pachyderm, you need to put data into it. Then, data is transformed through a number of transformation steps. After that, you can export your data and models to an outside source in a form of libraries, binaries, packages, tables, dashboards, or any other format for further use. In this section, we will review the ways to upload and download data to and from Pachyderm and all the native Pachyderm modifications that can be applied during this process.

Let's begin with uploading data to Pachyderm.

Uploading data to Pachyderm

You can divide data sources that ingest data into Pachyderm into the following categories:

From a local filesystem

From a URL

A remote object or block storage

A streaming or messaging platform

In this section, you will likely mostly use your local filesystem to upload data to Pachyderm repositories. This can be done with a simple Pachyderm command:

To upload data from a local filesystem, you use the following:

pachctl put file -f <filename> repo@branch

The repository must exist.

Similarly, to upload the data stored in a remote location, you can use the same command by specifying the correct URL:

pachctl put file -f https://mylink repo@branch

The same applies to the files stored in an object store. For example, if you are using an object store on Google Cloud Platform, you use the following:

pachctl put file -f gs://my-bucket repo@branch

The preceding commands put the files in the root of the repo, but you could also put them in any subdirectory by specifying the path to them, like this:

pachctl put file -f gs://my-bucket repo@branch:/path

To upload a whole directory, use the -r flag:

pachctl put file -f directory -r repo@branch:/

Run pachctl put file --help to view more examples.

If you want to automatically upload data from a messaging queue to Pachyderm, you need to create a special type of pipeline called a spout. The main thing a spout does is that it connects a data stream from a messaging platform to Pachyderm and puts it into a specified Pachyderm repository. You could also specify some code to categorize data into folders according to the preferred pattern. Here is an example of a spout pipeline:

---

pipeline:

  name: my-spout

spout: {}

transform:

  cmd:

  - python

  - myspout.py

  image: registry/image:1.0

  env:

    HOST: my-messaging-queue

    TOPIC: mytopic

    PORT: '5672'

Now that we know how we can put data into Pachyderm, let's take a look at data provenance and data lineage in Pachyderm.

About data lineage

If your system relies on data, you need to ensure that the data you use in your decision-making process is accurate and credible. Failure to provide a traceable data footprint may result in negative consequences for your organization. As more and more data-based systems are used in all aspects of our lives, wrong decisions based on erroneous data can have devastating impacts on people's lives.

That's why being able to go back in time and trace the data to its origins is a crucial part of any data management system. The ability to track the changes that happened to the data through the multiple transformation steps back to its origin is called data lineage or data provenance.

Typically, data lineage is visualized in the form of a Direct Acyclic Graph (DAG). Here is an example of the DAG representation in the Pachyderm UI:

Figure 7.2 – Pachyderm DAG

Each container represents either an input or output repository or a pipeline. The preceding example is very simple. In a workflow with more steps, the DAG might look more complex.

Why is data lineage so important? Here are a few important points to consider:

With topics such as privacy and equality being widely discussed on all levels, providing an audit trail for governance is becoming a necessity.

With data science being widely productized, implementing a system that offers the ability to roll back and correct errors in data and algorithms is crucial.

Trustworthiness of the data is another aspect that can be verified through data lineage.

In Chapter 1, The Problem of Reproducibility, we discussed many examples where the lack of a proper data management system can have a devastating impact on people's lives, as well as harming your businesses.

Now that we have discussed the importance of data lineage, let's take a look at how you can explore data lineage in Pachyderm.

Exploring data lineage

Data provenance or data lineage is the most important feature of Pachyderm that ensures that your changes are preserved and can be traced to the beginning of the pipeline's existence.

To demonstrate this functionality, we will use the same pipeline we used in Chapter 6, Creating Your First Pipeline. If you have not downloaded the files yet, go to https://github.com/PacktPublishing/Reproducible-Data-Science-with-Pachyderm/tree/main/Chapter06-Creating-Your-First-Pipeline and download them from there:

Create the Pachyderm photos repository, put the brown_vase.png file in it, and create the contour and histogram pipelines by running the following commands:

pachctl create repo photos

pachctl put file -f brown_vase.png photos@master

pachctl create pipeline -f contour.yaml

pachctl create pipeline -f histogram.yaml

You should see the following output:

brown_vase.png 25.82KB / 25.82 KB [================] 0s 0.00 b/s

The following command shows how you can view the lineage of your data and pipelines. First, we need to get a commit number for the desired data change:

pachctl list commit contour@master

The output should look similar to this:

REPO  BRANCH COMMIT  FINISHED SIZE ORIGIN DESCRIPTION

contour master 3d42... 22 seconds ago 23.78KiB AUTO

In this example, we only have one output commit with the 3d42e6385854478fbd2c9212c3afdab2 hash.

Then, we can run the inspect commit command to get the provenance information:

pachctl inspect commit contour@3d42e6385854478fbd2c9212c3afdab2

The preceding command returns the following output:

{

  "commit": {

    "branch": {

      "repo": {

        "name": "contour",

        "type": "user"

      },

      "name": "master"

    },

    "id": "3d42e6385854478fbd2c9212c3afdab2"

  },

  "origin": {

    "kind": "AUTO"

  },

  "child_commits": [

    {

      "branch": {

        "repo": {

          "name": "contour",

          "type": "user"

        },

        "name": "master"

      },

      "id": "dfff764bd1dd41b9bf3613af86d6e45c"

    }

  ],

  "started": "2021-08-18T17:03:32.180913500Z",

  "finishing": "2021-08-18T17:03:39.172264700Z",

  "finished": "2021-08-18T17:03:39.225964100Z",

  "direct_provenance": [

    {

      "repo": {

        "name": "contour",

        "type": "spec"

      },

      "name": "master"

    },

    {

      "repo": {

        "name": "photos",

        "type": "user"

      },

      "name": "master"

    }

  ],

  "size_bytes_upper_bound": "24353",

  "details": {

    "size_bytes": "24353"

  }

}

This output shows you that the commit is created in the photos repository. It has the AUTO type because it was generated when the data was uploaded to the photos repository. You can also see that it has created a child commit, dfff764bd1dd41b9bf3613af86d6e45c, for which you can run the same command. The child commit will have the ALIAS type since it is connected to the original commit in the photos repository. Over time, as new data arrives, this list will grow.

The preceding command tells us what the predecessors of the commit in the contour repository were. We can also use the wait commit command with a commit ID to track commits as they land in the output repository. For example, if we want to watch the changes that the 3e16201310374944b48ed73f83be2be7 commit lands in downstream pipelines, we can run the following command:

pachctl wait commitset photos@438428d0c3a145aa905c86c9fb1789ea

Provenance is a powerful feature of Pachyderm. It is especially useful when you need to find an audit trail to find out what made your pipeline biased.

Now that we have learned how to explore data provenance in Pachyderm, let's look into how to mount your Pachyderm repository to a local filesystem.

Mounting a Pachyderm repository to a local filesystem

You can mount your Pachyderm system to your local computer by using the Filesystem in Userspace (FUSE) interface to access your Pachyderm repositories as you typically would access local files. FUSE is supported on all major platforms, such as Microsoft Windows, Linux, and macOS. By default, you can mount your Pachyderm repositories with read-only access but write access can also be enabled. You need to understand that modifying the files in these mounts leads to broken provenance and should not generally be used. Use this functionality to do the following:

View the results of your pipeline runs.

Give third-party applications access to Pachyderm output repositories for further processing.

To mount a Pachyderm repository to your local computer filesystem, complete the following steps:

Install FUSE on your machine. If you are on macOS, run the following:

brew install osxfuse

If you are on Linux, run the following:

sudo apt-get install -y fuse

On Windows, run the following:

choco install winfsp

Use the pachctl mount command to mount a Pachyderm repository. For example, to mount a contour repository, run the following:

pachctl mount ~/Documents/contour --repos contour@master

This command will run continuously in your terminal until you interrupt it with Ctrl + C.

To access your files, go to your file browser. For example, if you are on macOS, use Finder. You should see the ~/Documents/contour folder mounted like this:

Figure 7.3 – Mounted Pachyderm repository

From here, you can view the files as needed.

To mount a repository with write access, run the same command with the --write flag:

pachctl mount ~/Documents/contour --repos contour@master --write

Use this functionality with caution as modifying files in output repositories breaks the provenance.

To mount multiple repositories, specify the list of the repositories each with a --repos flag:

pachctl mount ~/Documents/pachyderm-repos --repos contour@master --repos data@master --repos histogram@master --repos photos@master

The following screenshot shows how the data, contour, histogram, and photos repositories are mounted on your machine:

Figure 7.4 – Mounted Pachyderm repositories

In this section, we learned how to perform the most common Pachyderm data operations, including uploading data to Pachyderm, exploring provenance, and mounting Pachyderm repositories to your local machine, as well as splitting data while uploading it to Pachyderm. Next, we'll look into the most common pipeline operations that you will have to perform while working with Pachyderm.

Executing pipeline operations

Apart from creating and deleting pipelines, you likely will need to update your pipelines with new code changes. If changes are made to the pipeline specification itself, such as increasing the number of Pachyderm workers, input repository, glob pattern, or similar, you only need to do it in the YAML or JSON file and update the version of your pipeline spec. However, if the changes are in your code and your code is in your Docker image, you need to rebuild the Docker image. Let's go through each of these use cases.

Updating your pipeline specification

The pipeline specification enables you to control various Pachyderm parameters, such as controlling from which repository your pipeline consumes data, how many workers are started, and how many resources are available to your pipeline. You can also specify your code in the pipeline itself through the stdin field. Such a pipeline can use a basic Docker image that you won't have to ever update. If this is your case and you need to make changes in your pipeline spec or the code in the stdin field, here is what you need to do:

Make the needed changes in the YAML or JSON file that has your pipeline specification. For example, if you want to change a glob pattern in your contour.yaml pipeline from glob: "/" to glob: "/*", you just make these changes directly in the YAML file and save it.

Then, you need to update the version of the already running pipeline by specifying this YAML file:

pachctl update pipeline -f contour.yaml

Verify that the pipeline version is updated:

pachctl list pipeline

If the previous version of your pipeline was 1, it should change to 2:

  NAME      VERSION INPUT     CREATED       STATE / LAST JOB  DESCRIPTION

contour   2       photos:/* 6 seconds ago running / success A pipeline that identifies contours on an image.

The new pipeline will not process the data that has already been processed unless you explicitly specify it by using the --reprocess flag.

To run the updated pipeline against the already processed data, use the following:

pachctl update pipeline -f contour.yaml --reprocess

In the output, you should see that the version was updated again:

NAME      VERSION INPUT     CREATED        STATE / LAST JOB  DESCRIPTION

contour   3       photos:/* 15 seconds ago running / success A pipeline that identifies contours on an image.

By default, if you run pachctl list pipeline, Pachyderm shows information about the latest pipeline only. If you run this command with the --history flag, you can see all previous versions as well:

pachctl list pipeline contour --history 3

Here is what the output should look like:

NAME    VERSION INPUT    CREATED        STATE / LAST JOB  DESCRIPTION

contour 3       photos:/ 25 seconds ago running / success A pipeline that identifies contours on an image.

contour 3       photos:/ 25 seconds ago running / success A pipeline that identifies contours on an image.

contour 3       photos:/ 25 seconds ago running / success A pipeline that identifies contours on an image.

You can see that the version 3 pipeline ran three times.

Now, let's check that the third version of the pipeline output a commit in the contour repository:

pachctl list commit contour@master

This command should return similar output:

REPO    BRANCH COMMIT  FINISHED  SIZE  ORIGIN   PROGRESS DESCRIPTION

contour master 38eb403e62844f45939c6307bb0177c7 46 seconds ago 23.78KiB AUTO

Now that we know how to update a pipeline without changes to the code, let's see the workflow when your code is in a Docker image, and you need to update that code.

Updating your code

If your code is specified in a file that is embedded in a Docker image, you need to rebuild this Docker image every time you make changes to it, upload it to the Docker registry with a new version, update the version of the image inside of your pipeline specification, and then run the pachctl update pipeline command.

Let's modify the contour.py file in the contour pipeline that we created in Chapter 6, Creating Your First Pipeline. You need to have an account in a Docker registry to complete this section. If you do not have an account, you can create a free account on Docker Hub. All images referenced in this book are stored in Docker Hub and we will use Docker Hub as an example. If you are using any other Docker image registry, follow that registry's documentation to log in and upload images.

We will also need the Dockerfile for this pipeline to build new images:

If you have not done so yet, download the Dockerfile from https://github.com/PacktPublishing/Reproducible-Data-Science-with-Pachyderm/tree/main/Chapter06-Creating-Your-First-Pipeline.

Configure your Docker Hub account by signing up at https://hub.docker.com/.

When logged in to Docker Hub, click Create Repository.

Name your repository and write a short description. For this example, we will be using a public registry, but Pachyderm supports private registries as well:

Figure 7.5 – Creating a Docker Hub registry

Click Create.

Go to the terminal on your machine and run the following:

docker login

Provide your Docker Hub credentials to log in to Docker Hub. You should see the following output:

Login Succeeded

Open the contour.py file for editing.

Let's make a very small change in our code—on line 18, change linewidth to 1. Here is how it should look:

ax.plot(contour[:, 1], contour[:, 0], linewidth=1)

Save your changes and exit.

For these changes to take effect in our Pachyderm pipeline, we need to build a new Docker image. You need to have the Dockerfile in the current directory for the next command to work. Run the following command:

docker build . -t <your-registry>/contour-histogram:1.1

Replace <your-registry> with the name of your Docker Hub repository that you have created by following the preceding steps. You should see output similar to the following text:

Sending build context to Docker daemon 2.325GB

Step 1/10 : FROM ubuntu:18.04

---> 3339fde08fc3

…

Step 9/10 : ADD contour.py /contour.py

---> 4fb17a5f1f0b

Step 10/10 : ADD histogram.py /histogram.py

---> e64f4cb9ecb1

Successfully built e64f4cb9ecb1

The first time you build a Docker image, it might take some time. Note that in Step 9 in the preceding output, Docker adds your updated contour.py script.

The next step is to upload your image to your Docker registry. You could also first mount your Docker image locally and test it before uploading. To mount your Docker image locally, run the following:

docker save <your-registry>/contour-history:1.1 | (\

  eval $(minikube docker-env)

  docker load

)

This command takes some time to run, but it is very handy when you need to test something without constantly pushing new images to Docker Hub. We recommend that you mount your image locally, run your pipeline, and when ready, upload it to Docker Hub.

Or, if uploading directly to Docker Hub, run the following:

docker push <your-registry>/contour-history:1.1

After you load your image, you need to update your pipeline specification with the new version of your pipeline. Assign a new version number every time you build a new image so that you keep track of your changes. To update the image version in the pipeline specification, open the contour.yaml file and change the following online number 9:

9  image: <your-registry>/contour-histogram:1.1

Now, you can run your updated pipeline with the new code. Let's run the update command with the --reprocess flag to see how our changes have affected the result:

pachctl update pipeline -f contour.yaml --reprocess

In the following screenshot, you can see the comparison between the two versions. We have the first version on the left with a visibly thicker contour than the new version on the right:

Figure 7.6 – Comparing the results of the pipeline version

We have learned how to update Pachyderm pipelines. This method works for any language or framework. Pachyderm also provides built-in Docker build and Docker push commands that you could use. However, we suggest that you follow the process described previously as it seems to be more familiar to the majority of engineers, as well as more transparent.

Running maintenance operations

Like with every system or tool, Pachyderm might require periodic maintenance, upgrades, and troubleshooting. In the following sections, we will discuss the most important aspects of pipeline maintenance.

Troubleshooting your pipeline

In this section, you will learn how to troubleshoot your pipeline.

Your pipelines might fail for the following reasons:

Error in your code: This type of error occurs when something in your code is incorrect, such as a resource is not available or an incorrect value is specified. Fixing this type of error involves troubleshooting your code. You could try to do it locally before testing it inside Pachyderm.

Pipeline specification error: This type of error occurs when something is incorrect in the pipeline specification; for example, a pipeline cannot pull the Docker image. This often happens when a wrong image version is specified or there is a network issue.

Resource-related error: This type of error occurs when your cluster runs out of memory or has a similar resource constraint issue.

Pachyderm provides built-in functionality for pipeline troubleshooting through the pachctl logs command and you could also use Kubernetes-native tools to troubleshoot your pipelines. Since each Pachyderm pipeline is a Kubernetes Pod, you can use Kubernetes logging and debugging tools to troubleshoot them.

To detect and troubleshoot Pachyderm pipeline errors, complete the following steps:

When a pipeline fails, you should see it in the pipeline status:

pachctl list pipeline

Here is an example output of a failed pipeline:

NAME    VERSION INPUT    CREATED        STATE / LAST JOB    DESCRIPTION

contour 1       photos:/ 28 seconds ago crashing / starting A pipeline that identifies contours on an image.

To troubleshoot this pipeline, we need to view the logs for the pipeline or the job. To view the logs for the pipeline, run the following:

pachctl logs --pipeline=contour

The following is an example response:

container "user" in pod "pipeline-contour-v1-fmkxj" is waiting to start: image can't be pulled

In the preceding example, the failure is pretty clear—the pipeline failed to pull the Docker image. This could be due to the wrong image version specified in the pipeline spec or a network issue. Verifying that the pipeline spec is correct will likely solve the problem.

The following text is an example of another common code error in the pipeline:

Traceback (most recent call last):

  File "/pos-tag.py", line 13, in <module>

    with open('/pfs/out/pos-tag/pos-table.txt', 'w') as f:

FileNotFoundError: [Errno 2] No such file or directory: '/pfs/out/contour/pos-table.txt'

In the preceding example, the pipeline was not able to find a specified file. This is likely because the path to the file was specified incorrectly in the pos-tag.py file.

In some cases, you won't be able to see the job logs because the pipeline crashed before kicking off a job. However, in most cases, you should be able to view the logs for a job. To view the job logs, first get the hash of the job:

pachctl list job

Here is an example output:

ID                               SUBJOBS PROGRESS CREATED       MODIFIED

5865a26e1795481d96ecf867075c4f35 1       2 minutes ago 2 minutes ago

When you have a pipeline error, such as in the preceding output, the progress bar is yellow instead of green. This indicator gives you a clue that something is wrong in your code.

Use the job hash to view more information about the failure in the job logs:

pachctl logs --job=contour@5865a26e1795481d96ecf867075c4f35

The output should give you more information about the failure.

Finally, you can use kubectl to analyze possible errors in your pipeline. Get the name of your pipeline Pod by running the following:

kubectl get pod

You should see a similar response to the following:

NAME          READY   STATUS        RESTARTS   AGE

etcd-0         1/1     Running      0           6h10m

pachd-85d69d846-drgrk    1/1  Running 0         6h10m

pg-bouncer-84df8bdc58-7kzzg 1/1 Running 0       6h10m

pipeline-contour-v1-7dgwl 1/2  ImagePullBackOff 0    6m54s

postgres-0     1/1     Running      0           6h10m

You need to get logs for the pipeline Pod.

Get the Pod logs by running the following:

kubectl describe pod pipeline-contour-v1-7dgw

Important note

The Events part of the Pod logs typically provides information about any issues. Read more about Kubernetes debugging and troubleshooting in the Kubernetes documentation at https://kubernetes.io/docs/tasks/debug-application-cluster/.

This is an example output that you will see:

...

Events:

...

Normal BackOff 3m7s (x20 over 8m6s)  kubelet, minikube Back-off pulling image "svekars/contour-histogram:1.2"

In this section, we have discussed basic troubleshooting operations. The best strategy is to get as many logs as possible, categorize the problem, and then troubleshoot accordingly. If the problem is in the user code, you likely will want to test your code locally before running it in Pachyderm. One limitation that has been recently introduced in Pachyderm is the limit of the number of pipelines that you can run in the free tier. You won't be able to run more than 16 pipelines and 8 workers unless you upgrade to a paid version.

Next, we'll look into how to upgrade your cluster from one version to another.

Upgrading your Pachyderm cluster

Pachyderm releases minor version upgrades on a regular basis. Upgrading between minor versions and patches, such as from version 1.13.0 to 1.13.1 or 1.12.4. to 1.13.0, is pretty straightforward, but moving between major versions, such as 1.13.0 to 2.0, might be a bit more disruptive. Let's review the process for each of these use cases. Upgrades to major versions do not happen often. Typically, Pachyderm releases a major version once every few years. Those types of upgrades involve breaking changes and likely will have specific instructions. Refer to the Pachyderm documentation for steps to perform a major upgrade.

Upgrading between patches and minor versions

When you are upgrading your Pachyderm cluster, you need to make sure you back up your data and pipelines, upgrade the version of pachctl, and then redeploy your cluster. If you are upgrading locally in a minikube environment, you might not need to use your backup, but create one for safety reasons. If you are redeploying into the same namespace, all your data should still be available. If you are using a cloud environment, then you'll need to redeploy in a new namespace.

To upgrade from one patch or a minor version to another, complete the following steps:

Stop all pipelines that are currently running in your cluster:

pachctl stop pipeline <pipeline>

Let's stop the contour pipeline:

pachctl stop pipeline contour

Verify that the pipeline is suspended:

pachctl list pipeline

You should see the following output:

NAME      VERSION INPUT     CREATED      STATE / LAST JOB  DESCRIPTION

contour   1       photos:/* 3 hours ago paused / success  A pipeline that identifies contours on an image.

If you have any other pipelines running, stop them as well.

You need to make sure that all external data loading operations are suspended as well, if you have any automated services uploading data to Pachyderm.

Just in case, save your pachd, etcd, and console service manifests to separate YAML files:

kubectl get svc/pachd -o yaml > pachd_backup.yaml

kubectl get svc/etcd -o yaml > etcd_backup.yaml

kubectl get svc/dash -o yaml > dash_backup.yaml

If your upgrade goes wrong, you should be able to restore from these manifests manually.

Back up your cluster:

pachctl extract --no-auth --no-enterprise > my-pachyderm-backup

In the preceding example, we have specified the --no-auth and --no-enterprise flags. If you are using an enterprise version of Pachyderm or have enabled authentication, run this command without these flags.

Verify that the values.yaml file has updated values. Specifically, verify that the following autogenerated values are correct:

global:

      postgresql.postgresqlPassword

pachd:

      clusterDeploymentID

      rootToken

      enterpriseSecret

      oauthClientSecret

Upgrade your pachctl version, as in the following example:

brew upgrade pachyderm/tap/pachctl@2.0

Use the package manager in your system to upgrade.

Verify that the Pachyderm version was upgraded:

pachctl version --client-only

You should see the upgraded version in the output. In this case, it is 2.0.0:

pachctl             2.0.0

Use the helm upgrade command to redeploy your cluster:

helm upgrade pachd -f <pachyderm_deployment>_my_values.yaml pachyderm/pachyderm

Run kubectl get pod until you see the Pachyderm Pods running like this:

NAME                      READY   STATUS  RESTARTS AGE

console-5db94c4565-pzjft     1/1    Running   0     1m

etcd-0                       1/1    Running   0     1m

pachd-84984bf656-g4w8s       1/1    Running   0     1m

pg-bouncer-7f47f5c567-zwg8d  1/1    Running   0     1m

postgres-0                   1/1    Running   0     1m

Check that pachd is running:

pachctl version

This command should return an output similar to this:

COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

If you are in a cloud environment, use the following command to restore your configuration from the backup that you have created:

pachctl restore < my-pachyderm-backup

Check that your pipelines and repositories are in place:

pachctl list pipeline && pachctl list repo

The system response should look similar to this:

NAME    CREATED        SIZE (MASTER) DESCRIPTION

contour 49 seconds ago 23.78KiB      Output repo for ...

photos  49 seconds ago 25.21KiB

NAME    VERSION INPUT CREATED  STATE / LAST JOB DESCRIPTION

contour 1       photos:/* 6 minutes ago paused / success A pipeline that identifies contours on an image.

We have successfully restored our repositories and pipelines in our newly deployed cluster.

Cleaning up

After you are done experimenting, you might want to clean up your cluster so that you start your next experiment with a fresh install. To clean up the environment, run the following commands:

Delete all pipelines and repositories:

pachctl delete pipeline –all && pachctl delete repo --all

Verify that no repositories and pipelines exist in your cluster:

pachctl list repo && pachctl list pipeline

You should see the following output:

NAME CREATED SIZE (MASTER) DESCRIPTION

NAME VERSION INPUT CREATED STATE / LAST JOB DESCRIPTION

You have successfully cleaned up your cluster.

Summary

In this chapter, we have learned about some of the most important Pachyderm operations that you will need to perform during the lifetime of your Pachyderm cluster. We learned about the various ways to load data into Pachyderm, including how to do it with a messaging system. We learned how to update your pipelines, build Docker images, and mount them locally or upload them to a Docker image registry. Finally, we learned about some basic troubleshooting techniques and upgrading between patches and minor versions.

In the next chapter, we will implement an end-to-end machine learning workflow and learn more about deploying more complex multi-step Pachyderm pipelines.

Further reading

Docker Hub Quickstart: https://docs.docker.com/docker-hub/

Kubernetes monitoring, logging, and debugging: https://kubernetes.io/docs/tasks/debug-application-cluster/

Pachyderm build pipelines: https://docs.pachyderm.com/latest/how-tos/developer-workflow/build-pipelines/

Dua, D. and Graff, C. (2019). UCI Machine Learning Repository (http://archive.ics.uci.edu/ml). Irvine, CA: University of California, School of Information and Computer Science.

Chapter 8: Creating an End-to-End Machine Learning Workflow

In previous chapters, we learned about Pachyderm basics and how to install Pachyderm locally and on a cloud platform. We've deployed our first pipeline, learned how to update a pipeline, and performed some fundamental Pachyderm operations, such as splitting. I hope by now you are convinced that Pachyderm is an extremely versatile tool that gives you a lot of flexibility and power in managing your machine learning pipelines. To make it even more obvious, we will deploy a much more complex example than the ones that we have deployed so far. We hope this chapter will be especially fun for you to work on and will expand your understanding of data infrastructure quirks even more.

In this chapter, we will deploy a multistep Natural Language Processing (NLP) workflow that will demonstrate how to use Pachyderm at scale.

This chapter includes the following topics:

NLP example overview

Creating repositories and pipelines

Creating an NER pipeline

Retraining an NER model

Technical requirements

This chapter requires you to have the following components installed and configured.

For a local macOS installation, you need the following:

macOS Mojave, Catalina, Big Sur, or later

Docker Desktop for Mac 10.14

minikube v1.9.0 or later

pachctl 2.0.0 or later

Pachyderm 2.0.0 or later

For a local Windows installation, you need the following:

Windows Pro 64-bit v10 or later

Windows Subsystem for Linux (WSL) 2 or later

Microsoft PowerShell v6.2.1 or later

Hyper-V

minikube v1.9.0 or later

pachctl 2.0.0 or later

Pachyderm 2.0.0 or later

For an Amazon Elastic Kubernetes Service (Amazon EKS) installation, you need the following:

kubectl v.18 or later

eksctl

aws-iam-authenticator

pachctl 2.0.0 or later

Pachyderm 2.0.0 or later

For a Microsoft Azure cloud installation, you need the following:

kubectl v.18 or later

Azure CLI

pachctl 2.0.0 or later

Pachyderm 2.0.0 or later

jq 1.5 or later

For a Google Kubernetes Engine (GKE) cloud installation, you need the following:

Google Cloud SDK v124.0.0 or later

kubectl v.18 or later

pachctl 2.0.0 or later

Pachyderm 2.0.0 or later

The minimum virtual hardware requirements for a cloud or local virtual machine are as follows:

Number of CPUs: 4

Memory: 8,192 MB

Disk: 20 GB

Now that we know the technical requirements needed to accomplish the tasks in this chapter, we can deploy a Pachyderm instance with sufficient resources to run the example described in this chapter.

Adjusting virtual machine parameters

To run the example described in this section, you must make sure that the virtual machine that runs Pachyderm has enough memory and CPU to accommodate the pipeline requirements. This applies to both cloud and local environments.

If you are running Pachyderm on a cloud platform, make sure that you have deployed Kubernetes on a virtual machine flavor that adheres to the minimum hardware requirements listed in the Technical requirements section for this chapter. Then, redeploy your Pachyderm cluster as described in Chapter 5, Installing Pachyderm on a Cloud Platform.

If you are running Pachyderm in minikube on your local computer, make sure that the minikube virtual machine is large enough. If you have a minikube machine deployed as described in Chapter 4, Installing Pachyderm Locally, you need to delete it and deploy a new minikube virtual machine with a larger CPU and memory.

To do so, complete the following steps:

Uninstall the old Pachyderm cluster:

helm uninstall pachd

The system response is as follows:

release "pachd" uninstalled

Delete the existing minikube virtual machine:

minikube delete

You should see the following system response:

Deleting "minikube" in docker ...

Deleting container "minikube" ...

…

After your old machine is deleted, start a new virtual machine with the following parameters:

minikube start --cpus 4 --memory 8192

This command returns the following response:

…

Starting control plane node minikube in cluster minikube

Creating docker container (CPUs=4, Memory=8192MB) ...

…

Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Now, redeploy your Pachyderm cluster as described in Chapter 4, Installing Pachyderm Locally. For simplicity, here are the commands that you need to run:

helm install --set deployTarget=LOCAL pachd ./pachyderm

You should see the following system response:

NAME: pachd

LAST DEPLOYED: Thu Aug 19 13:03:36 2021

NAMESPACE: default

STATUS: deployed

REVISION: 1

...

Run the following commands to connect to pachd:

pachctl config import-kube local --overwrite

pachctl config set active-context local

Now that we have Pachyderm deployed with enough resources to run the example in this chapter, let's review the NLP pipeline that we will create.

NLP example overview

In this section, we will review the end-to-end machine learning workflow that will help us understand how to schedule it in Pachyderm.

To demonstrate this functionality, we will create an NLP pipeline that will perform various text optimizations against the The Legend of Sleepy Hollow short story book written by Washington Irving.

But first, let's review what NLP is and the typical stages that are involved in NLP.

Introduction to NLP

NLP is a machine learning technique that enables you to analyze natural text, namely speech or written text. This branch of artificial intelligence has existed for many years, but with the advancement of computer and internet technologies, it has found new implementations.

So, how can you use NLP in your business or academic research? There are many ways, but the most common ones include the following:

Speech recognition: Technology that enables computers to understand human voice and speech.

Chatbots: Software that can answer questions and learn from provided answers. Older chatbots were based on rules defined by standard software engineering techniques, meaning that they could not evolve and could only produce mediocre responses. Newer bots are much more advanced.

Machine translation: Technology that automates the task of translating text and speech from one language to another. The most common example is, of course, Google Translate.

Text extraction, summarization, and classification: A very needed technique in our world of information overload. NLP enables you to create pipelines that provide insights about a text, such as a summary of a research paper or information about the keywords used on a page.

Sentiment analysis: A famous technique that helps classify information by its positive or negative tone. The most famous implementation of this technology is the Gmail email classifier that sorts your email into three categories: Primary, Promotions, and Social emails.

These are the main examples of NLP. However, this list is not complete. NLP is also used in bioinformatics to analyze genetic data, in finance for understanding market events and trends, in healthcare to understand patient information, and in many other areas.

Now that we know the areas in which NLP is applicable, let's review the main phases that are involved in building an NLP pipeline.

Learning the NLP phases

As we discussed in the previous section, NLP is used to solve a variety of text- and speech-related tasks. Like with other areas of machine learning, when you need to solve an NLP problem, you need to build a pipeline. There are a few definitions of an NLP pipeline, but typically, phases of an NLP pipeline include the following:

Text preprocessing or cleaning: This stage includes operations such as word and sentence segmentation, tokenization, removal of stop words and punctuation, converting words to lowercase, and lemmatization or stemming.

Structure analysis: This stage goes deeper into analyzing what the text is about. It includes operations such as Part-of-Speech (POS) tagging, dependency parsing, and chunking.

Feature extraction: This stage deals with answering specific questions about your data and finding relationships between your text entities. It might include tasks such as Named Entity Recognition (NER) and Named Entity Disambiguation (NED) or linking and sentiment analysis.

Modeling: This stage is where you train your model on training data and test it to further put it into a production environment.

Depending on what your use case is, your pipeline might include all or some of these stages and they might also be in a different order.

The following diagram demonstrates an example NLP pipeline:

Figure 8.1 – NLP pipeline

Now that we know the phases of an NLP pipeline, let's look more closely at the example that we will implement.

Reviewing the NLP example

In our example, we will be using the text of The Legend of Sleepy Hollow by Washington Irving and in the end, we will create and train an NER pipeline that will help us to answer the question of who the main characters in the book are. To create this multistep workflow, we will need to create the following pipelines:

Data cleaning pipeline: This pipeline will download the text from a provided URL and clean it of any HTML tags, headings, and other irrelevant content. Then, it tokenizes the text, removes stop words and punctuation, and then does stemming and lemmatization.

POS tagging pipeline: This pipeline will add POS tags to the cleaned text from the previous pipeline based on the position of the words in the sentences and context.

NER pipeline: This pipeline will run a pretrained model against our text and attempt to label the results with the correct entities.

NER training pipeline: This pipeline will train a new NER model based on provided training data to correct the results of the first NER pipeline.

Improved NER pipeline: This pipeline will run the new NER model against our text and output the list of characters in the story to a text file.

Here is a diagram of our full NLP pipeline workflow:

Figure 8.2 – Pachyderm NLP pipeline

Now that we have reviewed the pipeline steps, let's create all the needed repositories and pipelines step by step.

Creating repositories and pipelines

In this section, we will create all the pipelines that we reviewed in the previous section. The six-step workflow will clean the data, apply POS tagging, perform NER, train a new custom mode based on the provided data, run the improved pipeline, and output the results to the final repo.

The first step is to create the data cleaning pipeline that will strip the text from the elements we won't need for further processing.

Important note

You need to download all files for this example from https://github.com/PacktPublishing/Reproducible-Data-Science-with-Pachyderm/tree/main/Chapter08-End-to-End-Machine-Learning-Workflow. The Docker image is stored at https://hub.docker.com/repository/docker/svekars/nlp-example.

Creating the data cleaning pipeline

Data cleaning is typically performed before any other types of tasks. For this pipeline, we have created a Python script that uses the Natural Language Toolkit (NLTK) platform to perform the data cleaning task. NLTK is an open source set of libraries that enables you to complete a variety of NLP-related tasks, including tokenization, stemming, removing stop words, and lemmatization.

Here is the pipeline specification that we will use for this pipeline:

---

pipeline:

   name: data-clean

description: A pipeline that tokenizes the text.

input:

   pfs:

     glob: "/data.txt"

     repo: data

transform:

   cmd:

   - python3

   - "/data-clean.py"

   image: svekars/nlp-example:1.0

This pipeline performs the following:

Takes a URL provided in the data.txt file from the data repository

Uses the svekars/nlp-example:1.0 image

Runs the data-clean.py script added to the svekars/nlp-example:1.0 image

You might have noticed that the glob pattern used in the pipeline specification uses only one file—data.txt. This is a file where we will have a URL to the The Legend of Sleepy Hollow text located at the Project Gutenberg: Free eBooks website. To access the website, go to https://gutenberg.org.

Now that we've reviewed the pipeline, let's look closer at what our script does. Here is a list of the components that we will be importing in the data-clean.py script:

from bs4 import BeautifulSoup

from urllib.request import urlopen

import nltk

from nltk.corpus import stopwords

from nltk.tokenize import word_tokenize

from nltk.stem.porter import PorterStemmer

nltk.download('wordnet')

from nltk.stem import WordNetLemmatizer

We need BeautifulSoup to parse the HTML file with our text. We use urlopen to open the URL inside of the data.txt file. We need NLTK with stopwords, word_tokenize, PorterStemmer, WordNet, and WordNetLemmatizer to perform various NLP operations.

The first part of code opens the data.txt file that we have placed in the data repository, reads the file, and uses the BeautifulSoup HTML parser to parse the text. In the paragraphs line, we strip the text from all other HTML elements but the <p> HTML tag:

with open('/pfs/data/data.txt', "r", encoding='utf-8') as f:

    data=f.read().replace('\n', '')

url = urlopen(data).read()

soup = BeautifulSoup(url, 'html.parser')

content = soup.get_text(strip=True)

paragraphs = soup.find_all('p')

The second part of the script saves the downloaded text to the text file in the output repository. We will need our downstream pipelines:

f = open('pfs/out/text.txt', 'w', encoding='utf-8')

for i in paragraphs:

     all_text = i.get_text()

      f.writelines(all_text)

f.close()

In the next part of the code, we use the word_tokenize NLTK method to break the text into individual tokens and save them into the tokens.txt file in the output repository:

tokens = []

for i in paragraphs:

    tokens += word_tokenize(i.text)

    with open('/pfs/out/tokens.txt', 'w', encoding='utf-8') as filehandle:

        for item in tokens:

            filehandle.write('%s\n' % item)

The next part of the code takes the tokenized text from previously, removes the stop words from it, and saves the result in the no_stopwords.txt file in the output repository. The stop words are the words that include articles, pronouns, and other commonly used words that don't add a lot of value to the text and can be ignored for research purposes to save processing time:

stopwords = stopwords.words("english")

no_stopwords = []

for word in tokens:

     if not word in stopwords:

         no_stopwords.append(word)

         appendFile = open('/pfs/out/no_stopwords.txt', 'a', encoding='utf-8')

         appendFile.write(word)

         appendFile.write("\n")

         appendFile.close()

The next part of the code removes punctuation from the text that was already tokenized and stripped of stop words. The code saves the results into a separate file called no_punctuation.txt:

no_punctuation = []

for word in no_stopwords:

      if word.isalpha():

          no_punctuation.append(word)

          appendFile = open('/pfs/out/no_punctuation.txt', 'a', encoding='utf-8')

          appendFile.write(word)

          appendFile.write("\n")

          appendFile.close()

Next is stemming. Stemming is a technique that removes suffixes from words leaving only the word stem. For example, the words grouping and grouped would be reduced to just group. Sometimes, this technique might be considered too aggressive and lemmatization can be used instead. The stemmed output is saved to stemmed.txt:

port_stem = PorterStemmer()

stemmed = []

for word in no_punctuation:

    stemmed_word = port_stem.stem(word)

    stemmed.append(stemmed_word)

    appendFile = open('/pfs/out/stemmed.txt', 'a', encoding='utf-8')

    appendFile.write(stemmed_word)

    appendFile.write("\n")

    appendFile.close()

The final part of this script is lemmatization, which uses NLTK's WordNet Lemmatizer database to perform lemmatization on the text that was tokenized and stripped of stop words and punctuation. This last piece of code saves the results to the lematized.txt file. We will use that file in our next pipeline:

lemmatizer = WordNetLemmatizer()

lemmatized = []

for word in no_punctuation:

    l_text = lemmatizer.lemmatize(word)

    lemmatized.append(l_text)

    appendFile = open('/pfs/out/lematized.txt', 'a', encoding='utf-8')

    appendFile.write(l_text)

    appendFile.write("\n")

    appendFile.close()

Now that we know what our pipeline does, let's create it.

To create the data-clean.py pipeline, complete the following steps:

Open your terminal and verify that Pachyderm is up and running:

pachctl version

The system output is as follows:

COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

Create the data repo:

pachctl create repo data

Verify that that data repository was created:

pachctl list repo

The system output is as follows:

NAME CREATED   SIZE (MASTER) DESCRIPTION

data 10 seconds ago ≤ 0B

From the directory where you have your data.txt file, put it into the data repository:

pachctl put file -f data.txt data@master

The system output is as follows:

data.txt 49.00b / 49.00 b [==============] 0s 0.00 b/s

This file has just one line—a link to the text of The Legend of Sleepy Hollow on the Gutenberg website: https://www.gutenberg.org/files/41/41-h/41-h.htm.

Check that the file was placed in the repository with the file type:

pachctl list file data@master

The system output is as follows:

NAME      TYPE SIZE

/data.txt file 49B

Create the data-clean pipeline:

pachctl create pipeline -f data-clean.yaml

No output is returned.

Check that the pipeline was created and is starting or running:

pachctl list pipeline

The system output is as follows:      

NAME       VERSION INPUT          CREATED       STATE / LAST JOB DESCRIPTION

data-clean 1       data:/data.txt 4 seconds ago running / -      A pipeline that tokenizes the text.

After a minute or so, the pipeline should finish running and upload the results to the output repository. Check the repository:

pachctl list repo

You should see the following output:

NAME       CREATED        SIZE (MASTER) DESCRIPTION

data-clean 15 seconds ago ≤ 315.8KiB    Output repo for pipeline data-clean.

data       1 minute ago ≤ 49B

As you can see, Pachyderm automatically created an output repository called data-clean and uploaded 315.8 KiB to the master branch of that repo.

Let's list the files in the repo:

pachctl list file data-clean@master

The system output is as follows:

NAME                TYPE SIZE

/lemmatized.txt     file 42.25KiB

/no_punctuation.txt file 42.99KiB

/no_stopwords.txt   file 47.88KiB

/stemmed.txt        file 38.01KiB

/text.txt           file 75.18KiB

/tokens.txt         file 69.54KiB

You can see that the pipeline has uploaded six files into the output repository. We have deliberately saved them in separate files so that you can see the difference between them. You can view the contents of each file and compare them. For example, to open the lemmatized.txt file on macOS, run the following:

pachctl get file data-clean@master:/lemmatized.txt | open -f  -a  TextEdit.app

You should see the following output:

Figure 8.3 – Lemmatized words

In this section, we have created a pipeline that cleans our text. In the next section, we will create our next pipeline, which will apply POS tags to our lemmatized text.

Creating the POS tagging pipeline

POS tagging is an NLP technique that labels each word with a relevant part of speech. This process is used in many NLP problems, such as text disambiguation and text-to-speech conversion.

For this task, we have used spaCy, a free library that performs POS tagging, NER, and other tasks. For example, say you have the following sentence:

Whoever is happy will make others happy too.

Here is an example of what POS tagging with spaCy looks like:

Figure 8.4 – POS tagging example

We will use spaCy to find POS tags in our lemmatized text from the data-clean pipeline.

Here is what our POS tagging pipeline specification will look like:

---

pipeline:

   name: pos-tag

description: A pipeline that performs POS tagging.

input:

   pfs:

     glob: "/lemmatized.txt"

     repo: data-clean

transform:

   cmd:

   - python3

   - "/pos-tag.py"

   image: svekars/nlp-example:1.0

This pipeline performs the following:

Takes the lemmatized.txt file from the data-clean repository

Uses the svekars/nlp-example:1.0 Docker image

Runs the pos-tag.py script against our lemmatized text

Outputs a table with all POS tags found in the text inthe pos_table.txt file, a file with the total number for each POS tag, to pos_number.txt and creates a dependency graph saved as a pos-tag-dependency.svg file

Now that we have reviewed what the pipeline does, let's take a look at the pos-tag.py script.

The script imports the following components and libraries:

import spacy

import en_core_web_sm

from spacy import displacy

import IPython

from pathlib import Path

import spacy.attrs

from contextlib import redirect_stdout

We need spacy and its modules to perform POS tagging, count them, and visualize the results. We import the en_core_web_sm pretrained spaCy model to do the tagging task. We need IPython as a spaCy dependency. Finally, we are using pathlib and redirect_stdout to save the results.

The first part of the code imports a pretrained spaCy model called en_core_web_sm. POS tagging requires you to use either a pretrained model or your own custom model. The en_core_web_sm model does a good job of tagging POSes. Therefore, we will just use it. Then, the script opens our lematized.txt file, tags all the words in the file, and prints out the result to the pos-table.txt file:

sp = spacy.load('en_core_web_sm')

textfile = sp(open('pfs/data-clean/lematized.txt', "r", encoding='utf-8').read())

with open('/pfs/out/pos-table.txt', 'w') as f:

     with redirect_stdout(f):

         for word in textfile:

             print(f'{word.text:{12}} {word.pos_:{10}} {word.tag_:{8}} {spacy.explain(word.tag_)}')

The next part of the code counts the number of each tag in the processed text and outputs the results to the pos-number.txt file:

with open('/pfs/out/pos-number.txt', 'w') as file:

     with redirect_stdout(file):

         count_tags = textfile.count_by(spacy.attrs.IDS['POS'])

         for i, count in count_tags.items():

             tags = textfile.vocab[i].text

             print(tags, count)

Finally, the last part of the script generates a dependency graph and saves it as an SVG image, pos-tag-dependency.svg:

image = displacy.render(textfile, style='dep', options={"compact": True, "distance": 70})

f = open('/pfs/out/pos-tag-dependency.svg', "w")

f.write(image)

f.close()

Now, let's create our pipeline.

To create a POS tagging pipeline, do the following:

Open your terminal and verify that Pachyderm is up and running:

pachctl version

You should see the following output:

COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

Create the POS tagging pipeline:

pachctl create pipeline -f pos-tag.yaml

No system output is returned.

Check that the pipeline was created and is running:

pachctl list pipeline

This is the output that this command returns:

NAME       VERSION INPUT   CREATED       STATE / LAST JOB  DESCRIPTION

pos-tag    1       data-clean:/lemmatized.txt 2 seconds ago running / -       A pipeline that performs POS tagging.

data-clean 1       data:/data.txt             1 minute ago running / success A pipeline that tokenizes the text.

When the pipeline finishes running, check the output repository:

pachctl list repo

This command returns the following output:

NAME       CREATED           SIZE (MASTER) DESCRIPTION

pos-tag    22 seconds ago  ≤ 10.82MiB    Output repo for pipeline pos-tag.

data-clean 2 minutes ago  ≤ 315.8KiB    Output repo for pipeline data-clean.

data       3 minutes ago ≤ 49B

Pachyderm has created an output repository called pos-tag and uploaded 10.82 MiB to the master branch of this repository.

Now, let's take a look at files that were uploaded to the output repo:

pachctl list file pos-tag@master

This command returns the following system output:

NAME                    TYPE SIZE

/pos-number.txt         file 132B

/pos-table.txt          file 564.1KiB

/pos-tag-dependency.svg file 10.27MiB

  Let's take a look at the number of each tag we have in our text:

pachctl get file pos-tag@master:/pos-number.txt

You should see the following output:

ADP 154

SPACE 6291

NOUN 3110

NUM 67

ADJ 789

PROPN 346

VERB 1053

ADV 374

DET 98

AUX 69

PRON 130

PART 9

SCONJ 59

CCONJ 15

INTJ 7

X 11

Finally, let's take a look at the dependency graph. If you are on macOS, run the following:

pachctl get file pos-tag@master:/pos-tag-dependency.svg > pos-tag-dependency.svg | open -f pos-tag-dependency.svg -a "Google Chrome"

Google Chrome opens the file:

Figure 8.5 – POS dependency graph

Important note

You might need to scroll down in your browser to see the graph. Because we ran POS tagging against the whole book, this graph is very long. You'll need to scroll horizontally to see it all.

In this section, we configured a POS tagging pipeline by using Pachyderm and spaCy, as well as visualizing it with a dependency graph. Next, we'll configure an NER pipeline that will help us find the main characters of the story.

Creating an NER pipeline

NER is an information extraction technique that recognizes entities in text and puts them in certain categories, such as person, location, and organization. For example, say we have the following phrase:

Snap Inc. Announces First Quarter 2021 Financial Results

If you use spaCy's en_core_web_lg against this phrase, you will get the following results:

Snap Inc. - 0 - 9 - ORG - Companies, agencies, institutions, etc.

First Quarter 2021 - 20 - 38 - DATE - Absolute or relative dates or periods

Name recognition can be useful in a variety of tasks. In this section, we will use it to retrieve the main characters of The Legend of Sleepy Hollow.

Here is what our NER pipeline specification will look like:

---

pipeline:

   name: ner

description: A NER pipeline

input:

   pfs:

     glob: "/text.txt"

     repo: data-clean

transform:

   cmd:

   - python3

   - "/ner.py"

    image: svekars/nlp-example:1.0

This pipeline performs the following:

Takes the original text of The Legend of Sleepy Hollow from the data-clean repository

Uses the svekars/nlp-example:1.0 Docker image

Runs the ner.py script

Outputs the results to the ner repository

Now, let's look at what the ner.py script does. Here is the list of components the script imports:

import spacy

from spacy import displacy

from contextlib import redirect_stdout

We need spacy to perform NER and the displacy module to visualize the results. redirect_stdout is a handy way to redirect printed output to a file.

The rest of the code imports spaCy's pretrained model called en_core_web_lg. This model seems to perform better on NER tasks than its counterpart that we used in the POS tagging pipeline. Then, the script grabs the original text stored in the text.txt file in the data-clean repository and performs the NER task:

sp = spacy.load("en_core_web_lg")

def display_entities(text):

     with open ('/pfs/out/ner-list.txt', 'w') as f:

         with redirect_stdout(f):

             if text.ents:

                 for i in text.ents:

                     print(i.text+' - '+str(i.start_char)+' - '+str(i.end_char)+' - '+i.label_+' - '+str(spacy.explain(i.label_)))

text = sp(open('/pfs/data-clean/text.txt', "r", encoding='utf-8').read())

display_entities(text)

Finally, the script visualizes the results with displacy and saves them in the HTML format:

with open ('/pfs/out/ner-labels.html', 'w') as f:

      with redirect_stdout(f):

          for i in text.ents:

              html=displacy.render(text, style="ent", page=True)

              print(html)

Now that we know what our script does, let's create the pipeline.

To create the NER pipeline, complete the following steps:

Open your terminal and verify that Pachyderm is up and running:

pachctl version

This command returns the following output:

COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

Create the POS tagging pipeline:

pachctl create pipeline -f ner.yaml

No system output is returned.

Check that the pipeline was created and is starting or running:

pachctl list pipeline

You should see the following output:

NAME       VERSION INPUT                     CREATED           STATE / LAST JOB  DESCRIPTION

ner        1       data-clean:/text.txt      5 seconds ago     running / running A NER pipeline

pos-tag    1       data-clean:/lematized.txt 1 minutes ago running / success A pipeline that performs POS tagging.

data-clean 1       data:/data.txt            2 minutes ago       running / success A pipeline that tokenizes the text.

When the pipeline finishes running, check the output repository:

pachctl list repo

The command returns this system output:

NAME       CREATED        SIZE (MASTER) DESCRIPTION

ner        36 seconds ago ≤ 43.49MiB    Output repo for pipeline ner.

pos-tag    2 minutes ago  ≤ 10.82MiB    Output repo for pipeline pos-tag.

data-clean 3 minutes ago ≤ 315.8KiB    Output repo for pipeline data-clean.

data       4 minutes ago ≤ 49B

Pachyderm has created an output repository called ner and uploaded 43.49 MiB to the master branch.

Let's take a look at the files that were uploaded to the output repo:

pachctl list file ner@master

The following output is returned:

NAME             TYPE SIZE

/ner-labels.html file 43.47MiB

/ner-list.txt    file 19.36KiB

This repository has two files. One is the list of all instances of entities found in the file and the other is the visualization for all the entities.

Print the first 10 lines of the ner-list.txt file to the terminal:

pachctl get file ner@master:/ner-list.txt | awk 'FNR <= 10'

This command returns the following output:

one - 36 - 39 - CARDINAL - Numerals that do not fall under another type

Hudson - 108 - 114 - LOC - Non-GPE locations, mountain ranges, bodies of water

Dutch - 186 - 191 - NORP - Nationalities or religious or political groups

the Tappan Zee - 203 - 217 - EVENT - Named hurricanes, battles, wars, sports events, etc.

St. Nicholas - 303 - 315 - ORG - Companies, agencies, institutions, etc.

Greensburgh - 417 - 428 - PERSON - People, including fictional

Tarry Town - 498 - 508 - GPE - Countries, cities, states

former days - 547 - 558 - DATE - Absolute or relative dates or periods

about two miles - 891 - 906 - QUANTITY - Measurements, as of weight or distance

first - 1330 - 1335 - ORDINAL - "first", "second", etc. NUM 67

As you can see, the NER model has identified many entities in the text. Let's open the HTML file to view all the entities.

Open the HTML file:

pachctl get file ner@master:/ner-labels.html > ner-labels.html | open -f ner-labels.html -a "Google Chrome"

The file will open in Google Chrome:

Figure 8.6 – NER labels

You can see that the spaCy model has identified many entities in the text correctly. However, if you start browsing, you'll notice that it has skipped some of them. For example, it did not tag Headless Horseman as PERSON in some cases. This is a known accuracy problem of pretrained models. In the next section, we will correct this by retraining our model to use these entities.

Retraining an NER model

Inaccuracy in NER pipeline results is a common problem. The only way to fix it is to retrain an existing model or train your own model completely from scratch. Training a model from scratch is a difficult and lengthy operation. In our case, we don't need to necessarily train a completely new model but instead, we can retrain the existing model to understand the missing context. To accomplish this task, we will put training data into the data-clean repository, create a training pipeline that will train on that data, save our model to an output repository, and then run the retrained model against our original text again.

In Pachyderm terms, this means that we will create two pipelines:

The first pipeline, called retrain, will train our model and output the new model to the train output repository.

The second pipeline, called my-model, will use the new model to analyze our text and upload the results to the my-model repository.

Now, let's create the retrain pipeline.

Creating the retrain pipeline

For this pipeline, we will create the following pipeline specification:

---

pipeline:

   name: retrain

description: A pipeline that retrains the NER model.

input:

   pfs:

     glob: "/training-data.json"

     repo: data

transform:

   cmd:

   - python3

   - "/ner-improved.py"

   image: svekars/nlp-example:1.0

This pipeline takes the training-data.json file that has our training data and runs the ner-improved.py script to improve the existing model. The results are saved to the retrain repository. For this example, we do not need a lot of training examples, but in a real-life use case, you would have to have hundreds of examples to improve the accuracy of your model.

Here is a list of the components the ner-improved.py script imports:

import spacy

import random

from spacy.util import minibatch

from spacy.training import Example

from contextlib import redirect_stdout

import simplejson as json

We need spacy with minibatch and Example.from_dict methods to train the model. We use random to shuffle the files into a different order for better training. The simplejson Python decoder is needed to read the training data file in the JSON format and redirect_stdout is needed to save the results in an output.

The next part of the script loads the spaCy model, reads the training data file, and opens a spaCy NER pipeline:

nlp=spacy.load("en_core_web_lg")

ner=nlp.get_pipe("ner")

data = open("/pfs/data/training-data.json")

data = json.loads(data.read())

This part of the code uses an optimizer, which performs the gradient descent calculation. Then, the script specifies that only the NER pipeline needs to be trained and all others should be ignored. The next for loop performs the actual training, updates the model, and prints the losses. We will train our model for 30 iterations:

optimizer = nlp.create_optimizer()

other_pipes = [p for p in nlp.pipe_names if p != "ner"]

with nlp.disable_pipes(*other_pipes):

     for i in range(30):

         random.shuffle(data)

         losses = {}

         for text, annotations in data:

             doc = nlp.make_doc(text)

             example = Example.from_dict(doc, annotations)

             nlp.update([example], drop=0.1, sgd=optimizer, losses=losses)

         print(losses)

The last part of the script tests the retrained pipeline against a test text and outputs the results to the ner-improved.txt text. The retrained model is saved using pickle.dump as in the ner-improved-model.p file directory in the output repository:

test_text = 'Headless Horseman came to see Ichabod Crane.'

doc = nlp(test_text)

with open ('/pfs/out/ner-improved.txt', 'w') as f:

    with redirect_stdout(f):

        for i in doc.ents:

            print(i.label_, " -- ", i.text)

pickle.dump(nlp, open('/pfs/out/ner-improved-model.p', 'wb'))

Now, let's create the pipeline:

Open your terminal and verify that Pachyderm is up and running:

pachctl version

The system output is as follows:

  COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

Put training-data.json in the data repository:

pachctl put file -f training-data.json data@master

Create the retrain pipeline:

pachctl create pipeline -f retrain.yaml

No system output is returned.

Check that the pipeline was created and is starting or running:

pachctl list pipeline

The system output is as follows:

NAME       VERSION INPUT                     CREATED        STATE / LAST JOB  DESCRIPTION

retrain    1       data:/training-data.json   About a minute ago running / success A pipeline that retrains the NER model.

ner        1       data-clean:/text.txt       2 minutes ago      running / success A NER pipeline

pos-tag    1       data-clean:/lemmatized.txt 3 minutes ago      running / success A pipeline that performs POS tagging.

data-clean 1       data:/data.txt             5 minutes ago      running / success A pipeline that tokenizes the text.

After some time, check the output repository:

pachctl list repo

The system output is as follows:

NAME       CREATED        SIZE (MASTER) DESCRIPTION

retrain    About a minute ago ≤ 821.8MiB    Output repo for pipeline retrain.

ner        2 minutes ago      ≤ 43.49MiB    Output repo for pipeline ner.

pos-tag    3 minutes ago      ≤ 10.82MiB    Output repo for pipeline pos-tag.

data-clean 5 minutes ago      ≤ 315.8KiB    Output repo for pipeline data-clean.

data       6 minutes ago      ≤ 205B

As you can see, Pachyderm uploaded 816.7 MiB to the retrain repository. This is our retrained model that takes this much space.

List the files in the retrain repo:

pachctl list file retrain@master

The system output is as follows:

NAME                TYPE SIZE

/ner-improved-model.p file 821.8MiB

/ner-improved.txt   file 56B

Let's look at the ner-improved.txt file, which should have the results of running the retrained model against the test text:

pachctl get file retrain@master:/ner-improved.txt

The system output is as follows:

PERSON  --  Headless Horseman

PERSON  --  Ichabod Crane

Great! Headless Horseman and Ichabod Crane are defined as PERSON.

Now that we have retrained our model, let's deploy our final pipeline, which will give us improved NER and output all the characters of the story into one file.

Deploying the retrained pipeline

Our retrained pipeline needs to be a cross-pipeline to combine our new retrained model with our text:

---

  pipeline:

    name: my-model

  description: A retrained NER pipeline

  input:

    cross:

    - pfs:

        repo: data-clean

        glob: "/text.txt"

    - pfs:

        repo: retrain

        glob: "/ner-improved-model.p"

  transform:

    cmd:

    - python3

    - "/ner-my-model.py"

    image: svekars/nlp-example:1.0

This pipeline will take the same text.txt file as in the original NER pipeline and create a cross-product of our retrained model with that text. It will output the results to the my-model repository. The resulting files will have an HTML file with better NER tagging and a text file with the list of characters in The Legend of Sleepy Hollow.

ner-my-model.py is very similar to the original ner.py script with the following differences.

It loads our improved model by using pickle.load instead of the original spaCy model:

nlp = pickle.load(open('/pfs/retrain/ner-improved-model.p', 'rb')))

It counts the total number of instances for each PERSON entity and saves them to the /pfs/out/person-label-count.txt file:

with open ('/pfs/out/person-label-count.txt', 'w') as f:

     with redirect_stdout(f):

         person_label=[]

         for i in text.ents:

             if i.label_ =='PERSON':

                 person_label.append(i.text)

         count = Counter(person_label)

         for key, counter in count.most_common():

It saves the HTML visualization to ner-improved-labels.html. It saves all entities to ner-improved-list.txt in the my-model repository.

Let's create our final pipeline:

Open your terminal and verify that Pachyderm is up and running:

pachctl version

The system output is as follows:

  COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

Create the my-model pipeline:

pachctl create pipeline -f my-model.yaml

This command does not return any output.

Check that the pipeline was created and is starting or running:

pachctl list pipeline

The system output is as follows:

NAME       VERSION INPUT                                                  CREATED        STATE / LAST JOB   DESCRIPTION

my-model   1       (data-clean:/text.txt ⨯ retrain:/ner-improved-model.p) 6 seconds ago  running / running A retrained NER pipeline

retrain    1       data:/training-data.json                               9 minutes ago  running / success A pipeline that retrains the NER model.

ner        1       data-clean:/text.txt                                   10 minutes ago running / success A NER pipeline

pos-tag    1       data-clean:/lemmatized.txt                             11 minutes ago running / success A pipeline that performs POS tagging.

data-clean 1       data:/data.txt                                         13 minutes ago running / success A pipeline that tokenizes the text.

Note that the my-model pipeline is starting. It is different from all other pipelines we've created for this example. Because we are saving our model in the retrain repository and we need to combine it with the text in the data repository, a standard pipeline won't be able to accomplish this. That's why we create a cross-pipeline that combines two inputs.

Let's check the output repository:

pachctl list repo

The system output is as follows:

NAME       CREATED        SIZE (MASTER) DESCRIPTION

my-model   About a minute ago ≤ 31.44MiB    Output repo for pipeline my-model.

retrain    10 minutes ago     ≤ 821.8MiB    Output repo for pipeline retrain.

ner        11 minutes ago     ≤ 43.49MiB    Output repo for pipeline ner.

pos-tag    12 minutes ago     ≤ 10.82MiB    Output repo for pipeline pos-tag.

data-clean 14 minutes ago     ≤ 315.8KiB    Output repo for pipeline data-clean.

data       15 minutes ago     ≤ 205B

Pachyderm uploaded 26.15 MiB to the my-model repository. This is the result of our computation.

List the files in the my-model repo:

pachctl list file my-model@master

The system output is as follows:

NAME                      TYPE SIZE

/ner-improved-labels.html file 26.14MiB

/ner-improved-list.txt    file 13.43KiB

/person-label-count.txt   file 654B

Let's look at the person-label-count.txt file, which should provide the total count for each unique PERSON instance:

pachctl get file my-model@master:/person-label-count.txt | awk 'FNR <= 15'

The system output is as follows:

Ichabod: 35

Brom Bones: 9

Ichabod Crane: 8

Van Tassel: 5

Hans Van: 5

Galloping Hessian: 4

André: 4

Headless Horseman: 3

Brom: 3

Hans Van Ripper: 3

Brouwer: 3

Tarry Town: 2

Cotton Mather: 2

Mather: 2

Baltus Van: 2

As you can see, the output is still not entirely accurate because we see instances of Ichabod and Ichabod Crane separately. If we provide more training data, we can improve these results. However, you already see the most often-listed characters in this list and can understand that Ichabod Crane is likely the main character of the story.

Open the HTML file to view the highlighted version of the results:

pachctl get file my-model@master:/ner-improved-labels.html > ner-improved-labels.html | open -f ner-labels.html -a "Google Chrome"

This concludes our experiment with the spaCy NER model. You can add more training data to see how the accuracy will improve with more training examples.

Now let's clean up our cluster.

Cleaning up

After you are done experimenting, you might want to clean up your cluster so that you start your next experiment with a fresh install. To clean up the environment, do the following:

Delete all pipelines and repositories:

pachctl delete pipeline –all && pachctl delete repo --all

Verify that no repositories and pipelines exist in your cluster:

pachctl list repo && pachctl list pipeline

You should see the following output:

NAME CREATED SIZE (MASTER) DESCRIPTION

NAME VERSION INPUT CREATED STATE / LAST JOB DESCRIPTION

You have successfully cleaned up your cluster.

Summary

In this chapter, we have learned how to build a complex machine learning workflow with the NER pipeline example. We have learned how to clean the data with the NTLK library, how to do POS tagging, and finally, how to retrain a spaCy model inside Pachyderm and output results for preview. You can do much more and tweak this example further to achieve better accuracy of NER by adding more training data and tweaking the model training parameters.

In the next chapter, we will learn how to do hyperparameter tuning in Pachyderm on an example of housing price prediction.

Further reading

NLTK documentation: https://www.nltk.org/

spaCy documentation: https://spacy.io/api/doc

The Legend of Sleepy Hollow on the Gutenberg project website: https://www.gutenberg.org/files/41/41-h/41-h.htm

Chapter 9: Distributed Hyperparameter Tuning with Pachyderm

In Chapter 8, Creating an End-to-End Machine Learning Workflow, we implemented an End-to-End (E2E) Machine Learning (ML) workflow based on a Named-Entity Recognition (NER) pipeline example. This was a multi-step pipeline that included many computational stages, including data cleaning, Part-Of-Speech (POS) tagging, model training, and running the new model against various data. Our goal was to find the main characters in the story, which we successfully achieved.

In this chapter, we will explore various strategies that can be implemented to select optimal parameters for an ML problem. This technique is called hyperparameter tuning or optimization. In the second part of this chapter, we will implement a hyperparameter tuning pipeline based on a house price prediction example.

This chapter includes the following topics:

Reviewing hyperparameter tuning techniques and strategies

Creating a hyperparameter tuning pipeline in Pachyderm

Technical requirements

This chapter requires you to have specific components installed and configured.

For a local macOS installation, you should have the following:

macOS Mojave, Catalina, Big Sur, or later

Docker Desktop for Mac 10.14

minikube v1.9.0 or later

pachctl 2.0.0 or later

Pachyderm 2.0.0 or later

For a local Windows installation, you should have the following:

Windows Pro 64-bit v10 or later

Windows Subsystem for Linux (WSL) 2 or later

Microsoft PowerShell v6.2.1 or later

Hyper-V

minikube v1.9.0 or later

kubectl v1.18 or later

pachctl 2.0.0 or later

Pachyderm 2.0.0 or later

For an Amazon Elastic Kubernetes Service (Amazon EKS) installation, you should have the following:

kubectl v.18 or later

eksctl

aws-iam-authenticator

pachctl 2.0.0 or later

Pachyderm 2.0.0 or later

For a Microsoft Azure cloud installation, you should have the following:

kubectl v.18 or later

The Azure Command-Line Interface (Azure CLI)

pachctl 2.0.0 or later

Pachyderm 2.0.0 or later

jq 1.5 or later

For a Google Kubernetes Engine (GKE) cloud installation, you should have the following:

Google Cloud Software Development Kit (SDK) v124.0.0. or later

kubectl v.18 or later

pachctl 2.0.0 or later

Pachyderm 2.0.0 or later

To be able to run the pipelines in this chapter, you do not need any special hardware. If you are running your Pachyderm cluster locally, any modern laptop should support all operations in this section. If you are running Pachyderm in a cloud platform, you will need to have a Persistent Volume (PV). See Chapter 5, Installing Pachyderm on a Cloud Platform, for more details.

All scripts and data described in this chapter are available at https://github.com/PacktPublishing/Reproducible-Data-Science-with-Pachyderm/tree/main/Chapter09-Distributed-Hyperparameter-Tuning-with-Pachyderm.

Now that we have reviewed the technical requirements for this chapter, let's take a closer look at our pipeline.

Reviewing hyperparameter tuning techniques and strategies

Hyperparameter tuning or hyperparameter optimization is a technique that ML professionals use to determine the best parameters to solve a specific ML problem. In different problems, you'd need to tune different types of parameters, such as weights in neural networks, or the number of trees in the Random Forest algorithm, or the learning rate of your model. Ultimately, selecting the best parameters helps you determine which method is best to solve a problem. A data scientist needs to understand the tunable parameters in the algorithm they use to be able to optimize them correctly.

There are a number of ML algorithms that help solve the hyperparameter optimization problem. Let's review the most common ones.

Grid search

Grid search is the simplest algorithm and is sometimes called a brute-force approach to hyperparameter optimization. This method calculates the optimum values of hyperparameters.

In Grid search, you typically define such hyperparameters as learning rate, dropout rate, or batch size. Then, you define a range of possible values. After that, the algorithm runs and searches for all possible configurations.

A disadvantage of Grid search is that it is computationally expensive and is typically used on a smaller set of hyperparameters.

Still, Grid search is a popular hyperparameter tuning algorithm and is the easiest one to understand.

The following diagram illustrates Grid search:

Figure 9.1 – Grid search

Now, let's look at another hyperparameter optimization technique called Random search.

Random search

Random search is similar to Grid search, but instead of checking all possible combinations, it selects them randomly, which often results in better performance and less computational time and resources. In many cases, Random search has proven to find the best combination faster than the Grid search method.

This diagram illustrates Random search:

Figure 9.2 – Random search

Now that we know what Random search and Grid search methods are, let's learn about a slightly more complex hyperparameter tuning method called Bayesian optimization.

Bayesian optimization

Bayesian optimization is a hyperparameter tuning technique that finds the minimum of a function. The main difference between Bayesian optimization and Grid search/Random search is that it keeps track of previous iterations and evaluation results and therefore uses probability (P) to predict the best combination.

A model trained with Bayesian optimization provides better results with more available data. Because it takes into account past results, such a model can find the best results with fewer iterations. Based on previous iterations, Bayesian optimization builds a posterior model that is closer to reality.

This diagram demonstrates the concept of Bayesian optimization:

Figure 9.3 – Bayesian optimization

We have learned about the three main hyperparameter optimization techniques. There are more available, but these three seem to be the most popular and widely used. Now, let's look into model evaluation metrics that we can use to determine the performance of our models. Because the problem we are going to discuss later in this chapter is a regression problem, we will only consider regression evaluation metrics.

Regression evaluation metrics

Just picking the right algorithm is only half of our success. We need to use an evaluation metric that determines the performance of our model. Evaluation metrics can be applied for various parameters to determine the best parameters. They can also be applied to multiple algorithms so that they can be compared and presented for further analysis.

Because the house price prediction example is a regression problem, we will only consider regression evaluation metrics. The most common evaluation metrics include the following:

R-squared (R2)

Mean Square Error (MSE)

Mean Absolute Error (MAE)

These metrics are known statistical methods for evaluating performance.

R2

R2 is an evaluation metric that is used in statistics to determine the variance in a dependent variable or how close the data is to the regression line. The parameter is measured as a percentage. If you get an R2 value of 100%, this means that the data fits the regression model perfectly. However, other values are acceptable, including 75%, 50%, and so on.

There are many ways to express R2 in a formula, but the simplest one looks like this:

If you are not that familiar with math, don't worry about this too much as in our code, we use the scikit-learn module to calculate R2.

Now that you have a general understanding of R2, let's learn about another evaluation metric called MSE.

MSE

MSE is an evaluation metric that estimates the difference between the predicted values with the observed value. MSE is highly influenced by outliers, values that are outside of the standard range. Therefore, you must remove outliers before you evaluate your model.

The lower the MSE value, the closer the result is to the real value.

For example, if you have a model that predicts salary based on years of service, your model MSE might be 200, meaning that the predicted value was US Dollars (USD) $200 higher than the actual value. Depending on the sample size, the overall scale, and expected precision, this number might have different significance.

The following formula is used to calculate MSE:

The following values are used in this formula:

n: A sample size

observed: The actual value

predicted: The value that the model has predicted

As with R2, don't worry about the formula too much as we will use the scikit-learn MSE module to calculate MSE.

Now, let's learn about MAE.

MAE

MAE is another evaluation metric often used with regression models. It calculates an average of total errors in your model or an absolute difference between real and forecasted values. If we were to express MAE in the simplest formula, it would look like this:

This metric is provided for your information only. We will not use it in our calculation. However, you can try it if you'd like by using the scikit-learn MAE module as well.

Apart from these metrics, there are a number of other metrics that can help you evaluate your model, including Root MSE (RMSE) and Adjusted R2. However, the ones that we mentioned previously are the most commonly used and the easiest to understand.

Now that we have learned about the methodology, evaluation metrics, and the algorithms we will use to configure hyperparameter tunning in Pachyderm, let's review the actual example, the model, the code, and the pipeline specifications. By the end of the next section, we will have a working hyperparameter tuning example in Pachyderm.

Creating a hyperparameter tuning pipeline in Pachyderm

In this section, we will explore our hyperparameter tuning pipeline and will create all the required attributes in Pachyderm to run our example.

Example overview

The house price prediction challenge is one of the classic ML examples of hyperparameter tunning optimization. It might not sound that complicated and may even be easy to predict based on your empirical experience. Likely, you know the area where you live pretty well and can estimate the price of houses based on square footage, number of rooms, adjacent land plot, and other parameters.

This information can be represented in a form of a two-dimensional (2D) array or a table with mentioned parameters. Here is an example of such a table:

Figure 9.4 – Sample housing data

Based on this information, you can predict the price of a house with similar characteristics without using any ML model and just by looking at these numbers.

But imagine that all the data you have is a Comma-Separated Values (CSV) file with thousands of rows and 60+ columns. You do not know anything about the area, and you haven't ever lived there. Imagine that you want to predict house prices continuously based on data that changes all the time. That's where creating a highly performing ML model comes in handy.

In our example, we will use a dataset that is available for free on Kaggle, a free repository with code and data. If you ever want a dataset to play with, you can find many examples on Kaggle. Our dataset is available at https://www.kaggle.com/lespin/house-prices-dataset/. We will only use the train.csv version of this dataset as we will modify it to clean the data.

The train.csv dataset includes 81 columns and 1,461 rows. You can view the data_description.txt file to review the column descriptions. The columns include the various parameters that affect the price of a house. Each row represents an example of a house sale with a specific sale price.

Here is an extract from the dataset:

Figure 9.5 – Extract from the housing dataset

We will attempt to create a model that we will train on our training data to predict house prices and will evaluate the performance of the model by using the R2 and MSE evaluation metrics that we discussed in the previous section.

The following diagram demonstrates our model:

Figure 9.6 – Hyperparameter tuning pipeline

Now that we have a basic idea of our pipeline, let's review each step of our pipeline workflow in more detail.

Creating an exploratory analysis pipeline

Our first pipeline explores the data and gives us some basic information about the dataset that we are using. Here is the pipeline specification of the exploratory analysis pipeline:

---

pipeline:

   name: data-explore

description: A pipeline that performs exploratory analysis.

input:

   pfs:

     glob: "/*"

     repo: data

transform:

   cmd:

   - python3

   - "/data-explore.py"

   image: svekars/hyperparameter-example:1.0

This pipeline takes all data from the data repository located under /* and runs the data-explore.py script against it. The pipeline uses the hyperparameter-example:1.0 Docker image.

Let's review what's the data-explore.py script does. The script imports the following components:

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns

We will use pandas to manipulate CSV tables and represent them as DataFrame structures. pandas is an open source Python library that is widely used by data scientists for data operations, specifically 2D tabular data.

We will use, familiar to us from previous sections, matplotlib in combination with the seaborn library to visualize the results of our computations. seaborn is based on matplotlib but provides more sophisticated and visually appealing graphs.

The first part of the data-explore.py script reads the housing-train.csv file as a DataFrame from the data repository and computes a correlation matrix between all columns in our dataset by using the pandas.DataFrame.corr() method. The code then creates a heatmap of the created correlation matrix and saves it in the pipeline output repository. The code is illustrated in the following snippet:

plt.subplots(figsize=(20,15))

data = pd.read_csv("/pfs/data/housing-train.csv", delimiter=',')

dataset = data.corr().round(2)

plt.subplots(figsize=(20,15))

my_plot = sns.heatmap(dataset, annot=True,cmap='YlGnBu', linecolor='white')

fig = my_plot.get_figure()

fig.savefig('/pfs/out/heatmap.png', dpi=400)

The second part of the code saves the types of data objects in a column. Typically, you use a different approach to analyzing and manipulating numerical and categorical data, which is why getting this information might be important. The code saves this information in a data-types.csv file and is illustrated here:

data_types = data.dtypes.to_frame('dtypes').reset_index()

data_types.to_csv('/pfs/out/data-types.csv', index=False)

The third part of the script checks the columns for missing data, creates a table with percentages of columns that have missing data, and saves the table to no-data.csv, as illustrated in the following code snippet:

cols_no_data = data.isnull().sum() / data.shape[0] * 100.00

no_data = pd.DataFrame({'Column': data.columns, 'Percentage Missing': cols_no_data})

no_data.sort_values(by=['Percentage Missing'], inplace=True, ascending=False)

header = ["Column", "Percentage Missing"]

no_data.to_csv('/pfs/out/no-data.csv', columns = header, index=False)

Let's create this pipeline, as follows:

Verify that Pachyderm is up and running by executing the following command:

pachctl version

This command returns the following output (your version of pachctl and pachd might be different):

COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

Create a data repository by running the following command:

pachctl create repo data

This command does not return any output.

Move the housing-train.csv file to the master branch of the data repository by running the following command:

pachctl put file data@master -f housing-train.csv

The system response should look like this:

housing-train.csv 460.68KB / 460.68 KB [=========] 0s 0.00 b/s

Verify that the file was added to the repository with the file type by running the following command:

pachctl list file data@master

The system response should look like this:

NAME               TYPE SIZE

/housing-train.csv file 449.9KiB

Create a data-explore pipeline by using the data-explore.yaml file, as follows:

pachctl create pipeline -f data-explore.yaml

This command does not return any response.

Verify that the pipeline was created by running the following command:

pachctl list pipeline

Here is the system response that you should see:

NAME      VERSION INPUT   CREATED  STATE / LAST JOB   DESC

data-explore 1    data:/* 34 seconds ago running / running A pipeline that performs exploratory data analysis.

Wait for the pipeline to finish running and display a success status for the last job.

List the repositories, as follows:

pachctl list repo

You should see that the data-explore pipeline uploaded 3.361MiB of data to the data-explore repository, as indicated in the following output:

NAME         CREATED        SIZE (MASTER) DESCRIPTION

data-explore 46 seconds ago 3.361MiB      Output repo for pipeline data-explore.

data         26 minutes ago 449.9KiB

Let's explore the data in the repository by running the following command:

pachctl list file data-explore@master

You should see the following three files:

NAME            TYPE SIZE

/data-types.csv file 1.37KiB

/heatmap.png    file 3.359MiB

/no-data.csv    file 1.447KiB

Let's open the /data-types.csv file, as follows:

pachctl get file data-explore@master:/data-types.csv

The file includes data types for each column, as we can see in the following code snippet:

index,dtypes

Id,int64

MSSubClass,int64

MSZoning,object

LotFrontage,float64

...

Let's look at what's in the no-data.csv file by running the following command:

pachctl get file data-explore@master:/no-data.csv

Alternatively, you could also open the file in an application on your computer. For example, in macOS, you can open it in the Numbers application, like this:

pachctl get file data-explore@master:/no-data.csv | open -f -a "Numbers"

This file contains information about columns and the percentage of missing data in these columns. This is very useful for data cleaning. Some columns have more than 80% of missing data. These columns can be removed so that they don't interfere with our calculations. We will do that in our next pipeline. Here is a list of columns with the majority of data missing:

Figure 9.7 – Columns with missing data

Now, let's look at the heatmap by running the following command:

pachctl get file data-explore@master:/heatmap.png | open -f -a "Preview.app"

You should see the following heatmap that shows us a correlation between all columns in the dataset:

Figure 9.8 – Heatmap of all parameters

This heatmap has too many parameters and is difficult to read. But even on this heatmap, we can see that some parameters affect the sale price more than others. For example, it looks as though the OverallQuality parameter affects the price the most, as well as the GrLivArea parameter (for great living area). We will try to narrow down our dataset to these parameters in our next pipeline step.

We have explored the dataset and got a basic understanding of the data. Now, let's review our next pipeline, which will clean up the data based on our findings.

Creating a data cleaning pipeline

Our next step is to create a pipeline that cleans our data. This pipeline will clean the data according to our findings in the previous section. Here is the pipeline specification:

---

pipeline:

   name: data-clean

description: A pipeline that removes empty cells from the CSV.

input:

   pfs:

     glob: "/"

     repo: data

transform:

   cmd:

   - python3

   - "/data-clean.py"

   image: svekars/hyperparameter-example:1.0

This is a standard Pachyderm pipeline that takes the data from the data repository and runs the data-clean.py script against that data. The data, in this case, is our housing-train.csv dataset.

Let's look at our data-clean.py script. The script imports the following components:

import pandas as pd

import matplotlib.pyplot as plt

import seaborn as sns

from pandas import Series

These components are similar to the ones in the data-explore pipeline. The new component that is being imported is pandas.Series, which we need to be able to save our data to a CSV file.

The first part of our script reads the data from the housing-train.csv dataset as a DataFrame. Then, we drop the columns that have more than 40% of columns with missing data and save the columns that we have dropped in the col_drop.csv file, as follows:

data = pd.read_csv("/pfs/data/housing-train.csv", delimiter=',')

col_drop = set(data.count()[data.count()<0.60*max(data.count())].index.tolist())

pd.Series(list(col_drop)).to_csv('/pfs/out/col_drop.csv', index=False)

Next, we create a new correlation matrix that only includes the parameters that affect the SalePrice column with a coefficient of 0.5 or larger. We plot a new heatmap and save it in the heatmap2.png file, as follows:

data = data.drop((col_drop), axis=1)

corr = data.corr()

r_var = corr.SalePrice[(corr.SalePrice > 0.5)]

r_col = list(r_var.index.values)

new_corr = data[r_col].corr()

plt.subplots(figsize=(20,15))

my_plot2 = sns.heatmap(new_corr, annot=True,cmap='YlGnBu', linecolor='white')

fig = my_plot2.get_figure()

fig.savefig('/pfs/out/heatmap2.png', dpi=400)

Finally, we remove the columns that are not part of the new correlation matrix and save them in a new dataset called cleaned-data.csv in the pipeline output repository, as follows:

new_data = data.loc[:, data.columns.intersection(r_col)]

new_data.to_csv('/pfs/out/cleaned-data.csv', index=True)

Now, let's create this data cleaning pipeline, as follows:

Verify that Pachyderm is up and running by executing the following command:

pachctl version

Here is the output:

COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

Your version of Pachyderm might be different.

Create a data-clean pipeline, as follows:

pachctl create pipeline -f data-clean.yaml

No system response is returned.

Verify that Pachyderm has successfully created the pipeline by running the following command:

pachctl list pipeline

Here is the system response that you should see:

NAME         VERSION INPUT   CREATED       STATE / LAST JOB  DESCRIPTION

data-clean   1       data:/ 6 seconds ago running / running A pipeline that removes empty cells from the CSV.

data-explore 1       data:/* 4 minutes ago running / success A pipeline that performs exploratory analysis.

You need to wait for the pipeline to change its status to success.

When the pipeline has successfully finished running, list the repositories by running the following command:

pachctl list repo

You should see that the data-clean pipeline added 780.4KiB of data, as indicated in the following output:

NAME         CREATED        SIZE (MASTER) DESCRIPTION

data-clean   42 seconds ago ≤ 780.4KiB    Output repo for pipeline data-clean.

data-explore 3 minutes ago  ≤ 3.361MiB    Output repo for pipeline data-explore.

data         12 minutes ago ≤ 449.9KiB

Let's look at the data in the repository by running the following command:

pachctl list file data-clean@master

The output should look like this:

NAME            TYPE SIZE

/cleaned-data.csv file 67.14KiB

/col_drop.csv     file 45B

/heatmap2.png     file 713.2KiB

Let's look at the columns that were dropped by running the following command:

pachctl get file data-clean@master:/col_drop.csv

The file includes data types for each column, as we can see here:

0

PoolQC

Alley

FireplaceQu

MiscFeature

Fence

These were the columns that had more than 40% columns empty.

We've also removed all the columns that we mapped in our new correlation matrix and that have a correlation coefficient of less than 0.5. Run the following command to see what our new correlation matrix looks like:

pachctl get file data-clean@master:/heatmap2.png | open -f -a "Preview.app"

Here is the new heatmap:

Figure 9.9 – Refined heatmap

This heatmap makes much more sense. We can clearly see the parameters that are the most prominent in determining the house sale price. Our dataset has been drastically reduced to 11 columns only and was saved to a new cleaned-data.csv file. Now, this strategy might not be ideal for every use case—you might decide to keep more parameters in your dataset to ensure and check if the model would perform better with more parameters. But for the purpose of this example, this should be sufficient.

Now that we have cleaned the data, we also need to make sure that we remove any outliers or parameters that are outside of the standard range. We will do this in our next section.

Creating a pipeline that removes outliers

Our next pipeline will evaluate outliers in our dataset and will remove them so that our model performance is not affected by them. We will again use a standard Pachyderm pipeline specification to achieve this goal. Here is the pipeline specification:

---

pipeline:

   name: remove-outliers

description: A pipeline that removes outliers from the dataset.

input:

   pfs:

     glob: "/"

     repo: data-clean

transform:

   cmd:

   - python3

   - "/outliers.py"

   image: svekars/hyperparameter-example:1.0

The pipeline specification takes the cleaned data from the data-clean repository and applies the outliers.py Python script to that data. The pipeline uses the same Docker image as the two previous ones.

The outliers.py script imports the same list of components as the scripts in our previous pipeline steps, including seaborn, matplotlib, and pandas.

The script reads the data from the cleaned-data.csv file. Then, it creates a histogram that displays outliers in the dataset and saves that histogram to the histogram.png file. Then, we only leave 50% of the data that is within the middle range and remove the rest. We create another histogram that shows this new data. We drop the data from the dataset and save it in a new CSV file called removed-outliers-data.csv. The code is illustrated in the following snippet:

data = pd.read_csv("/pfs/data-clean/cleaned-data.csv", delimiter=',', encoding='utf-8')

my_plot=sns.boxplot(x=data['SalePrice'])

fig = my_plot.get_figure()

fig.savefig('/pfs/out/histogram.png', dpi=400)

q1 = data['SalePrice'].quantile(0.25)

q3 = data['SalePrice'].quantile(0.75)

iqr = q3-q1

lw = q1 - 1.5*iqr

uw = q3 + 1.5*iqr

dataset = data[(data['SalePrice']>lw)&(data['SalePrice']<uw)]

plt.figure(figsize=(20,10))

my_plot2 = sns.histplot(data=dataset, x="SalePrice", color="orange", element="poly")

fig = my_plot2.get_figure()

fig.savefig('/pfs/out/histogram-outliers.png', dpi=400)

dataset.to_csv('/pfs/out/removed-outliers-data.csv', index=True)

Now, let's create this pipeline, as follows:

Verify that Pachyderm is up and running by executing the following command:

pachctl version

You should get an output similar to this:

COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

Your version might be different.

Create a data-clean pipeline, as follows:

pachctl create pipeline -f remove-outliers.yaml

This command does not return any output.

Check that the pipeline was created by running the following command:

pachctl list pipeline

Here is the system response that you should see:

NAME            VERSION INPUT         CREATED       STATE / LAST JOB   DESCRIPTION

remove-outliers 1       data-clean:/ 5 seconds ago running / starting A pipeline that removes outliers from the dataset.

data-clean      1       data:/       4 minutes ago running / success  A pipeline that removes empty cells from the CSV.

data-explore    1       data:/*       8 minutes ago running / success  A pipeline that performs exploratory analysis.

The remove-outliers pipeline is starting. You can run the pachctl list pipeline command several times until the pipeline succeeds.

List the repositories by running the following command:

pachctl list repo

The remove-outliers repository should have 413.7KiB of data uploaded to it, as indicated in the following output:

NAME            CREATED            SIZE (MASTER) DESCRIPTION

remove-outliers About a minute ago 413.7KiB      Output repo for pipeline remove-outliers.

data-clean      49 minutes ago     780.4KiB      Output repo for pipeline data-clean.

data-explore    53 minutes ago     3.361MiB      Output repo for pipeline data-explore.

data            About an hour ago  449.9KiB

List the files in the repository by running the following command:

pachctl list file remove-outliers@master

The output should look like this:

NAME                       TYPE SIZE

/histogram-outliers.png    file 290.8KiB

/histogram.png             file 52.8KiB

/removed-outliers-data.csv file 70.04KiB

Let's first open the histogram.png file by running the following command:

pachctl get file remove-outliers@master:/histogram.png | open -f -a "Preview.app"

Here is what you should see:

Figure 9.10 – Outliers in the dataset

This boxplot shows us that the majority of houses have a sale price in the range of $50,000 to $350,000, with most of these being in the range of $110,000 to $220,000. A few others are way outside of this range and might be even considered a separate category. Our pipeline removes outliers outside the main box.

Now, let's look at the histogram after the outliers were removed. We can do this by running the following command:

pachctl get file remove-outliers@master:/histogram-outliers.png | open -f -a "Preview.app"

Here is the new histogram:

Figure 9.11 – Histogram with removed outliers

We have removed some of the rows from our dataset and we now have 1,400 rows instead of the 1,481 rows that we had before.

Now that we have finished cleaning our data, we can next train our model.

Creating a training pipeline

Our next pipeline will train our model on a training part of our dataset. Here is the pipeline specification:

---

pipeline:

   name: train

description: A pipeline that trains the model with a selected estimator.

input:

   pfs:

     glob: "/"

     repo: remove-outliers

transform:

   cmd:

   - python3

   - "/train.py"

   image: svekars/hyperparameter-example:1.0

As you can see, this is another standard Pachyderm pipeline. It takes the data from the remove-outliers repository and applies the train.py script to it. It uses the same Docker image as other pipelines in this section.

Here is a list of components that the train.py script imports:

from sklearn.model_selection import train_test_split

from sklearn import metrics

import pandas as pd

from sklearn.metrics import r2_score, mean_squared_error, make_scorer

from sklearn.linear_model import Ridge

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns

from contextlib import redirect_stdout

We import train_test_split, metrics, r2_score, mean_squared_error, and make_scorer modules from the sklearn library to split the data into train and test data and calculate the R2 and MSE metrics for our model. We import the Ridge regression model from sklearn.linear_model to train our model using Ridge regression. Ridge regression is a variation of linear regression, one of the algorithms that you can use for this type of regression problem. We import seaborn and matplotlib to visualize our results, and pandas and numpy to manipulate the data. redirect_stdout is used to redirect output to a file.

The first part of our script reads the removed-outliers-data.csv file from the remove-outliers repository as a DataFrame. Then, we use train_test_split to split our dataset into training and testing parts. The training part is used to train the data, and the testing part is used to test the performance of our model in the cross-validation stage. The code is illustrated here:

data = pd.read_csv("/pfs/remove-outliers/removed-outliers-data.csv", delimiter=',')

X=data.drop('SalePrice', axis=1)

y=data['SalePrice']

train_X, test_X, train_y, test_y = train_test_split(X,y,test_size=0.4, random_state=0)

Next, we define our estimator, which is the Ridge regression. alpha is the parameter that we will be tuning to achieve better performance. We initially set alpha to 1, make a prediction, and save our R2 and MSE scores in the r_squared_mse.txt file, as illustrated in the following code snippet:

estimator = Ridge(alpha=1).fit(train_X, train_y)

prediction = estimator.predict(test_X)

with open('/pfs/out/r_squared_mse.txt', 'w', encoding='utf-8') as f:

     with redirect_stdout(f):

         print('R-squared:', metrics.r2_score(test_y, prediction))

         print('MSE:', np.sqrt(metrics.mean_squared_error(test_y, prediction)))

Lastly, we'll plot our data and save it in the prediction.png file, as follows:

plt.figure(figsize=(20,10))

myplot1 = sns.distplot(test_y, hist=True, kde=False)

myplot2 = sns.distplot(prediction, hist=True, kde=False)

plt.legend(labels=['Real Price', 'Predicted Price'])

plt.xlim(0,)

fig1 = myplot1.get_figure()

fig1.savefig('/pfs/out/prediction.png', dpi=400)

Let's create this pipeline, as follows:

Check that your Pachyderm cluster is running by executing the following command:

pachctl version

Here is the output:

COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

Create a train pipeline by running the following command:

pachctl create pipeline -f train.yaml

No output is returned.

List all pipelines by running the following command:

pachctl list pipeline

You should see the following output:

NAME            VERSION INPUT         CREATED       STATE / LAST JOB   DESCRIPTION

train           1       remove-outliers:/ 34 seconds ago running / success A pipeline that trains the model with a selected estimator.

remove-outliers 1       data-clean:/ 5 seconds ago running / starting A pipeline that removes outliers from the dataset.

data-clean      1       data:/       4 minutes ago running / success  A pipeline that removes empty cells from the CSV.

data-explore    1       data:/*       8 minutes ago running / success  A pipeline that performs exploratory analysis.

The output should list the train pipeline. Wait for the pipeline to finish running.

Let's look at the repositories. We can do this by running the following command:

pachctl list repo

You should see a new repository called train added with 186.3KiB of data, as indicated in the following output:

NAME            CREATED            SIZE (MASTER) DESCRIPTION

train           2 minutes ago 186.3KiB      Output repo for pipeline train.

remove-outliers About a minute ago 413.7KiB      Output repo for pipeline remove-outliers.

data-clean      49 minutes ago     780.4KiB      Output repo for pipeline data-clean.

data-explore    53 minutes ago     3.361MiB      Output repo for pipeline data-explore.

data            About an hour ago  449.9KiB

Now, let's look at the files that were uploaded to the repository. We can do this by running the following command:

pachctl list file train@master

The output should look like this:

NAME               TYPE SIZE

/prediction.png    file 186.2KiB

/r_squared_mse.txt file 54B

There should be two files.

Open the r_squared_mse.txt file to check the MSE and R2 scores. You can do this by running the following command:

pachctl get file train@master:/r_squared_mse.txt

The output should look like this:

R-squared: 0.7803812645495943

MSE: 29521.138357806965

Our R2 value is pretty high, meaning that the calculation should be quite precise.

Now, let's open the prediction.png file by running the following command:

pachctl get file train@master:/prediction.png | open -f -a "Preview.app"

This is what you should see:

Figure 9.12 – Predicted versus real price

As you can see, the predicted price looks pretty close to the real one, with a few minor exceptions.

In our last pipeline step, we will try to find the best value of alpha and will perform cross-validation of our parameters with Grid search.

Creating an evaluation pipeline

Our evaluation pipeline specification looks like this:

---

pipeline:

   name: evaluate

description: A pipeline that evaluates the performance of the model.

input:

   pfs:

     glob: "/"

     repo: remove-outliers

transform:

   cmd:

   - python3

   - "/grid-search.py"

   image: svekars/hyperparameter-example:1.0

Since you've seen quite a few of those, you can probably guess that it's a standard Pachyderm pipeline that takes data from the remove-outliers repository and applies the grid-search.py file to that data. The pipeline uses the same Docker image as all other pipelines.

The grid-search.py file imports the components already familiar to us from previous sections. In addition, it imports GridSearchCV from the sklearn.model_selection library and joblib, which saves the model to a pickle file.

The first part of the script performs the same data manipulation as in train.py—it opens the data file and splits it into two sets.

Next, we set the estimator property to have Ridge regression and specify scoring values and alpha parameters, like this:

estimator = Ridge(alpha=10)

scoring={'R_squared':'r2','MSE':'neg_mean_squared_error'}

params = {'alpha':[1,0.1,0.01,0.001,0.0001,0,10,100,1000]}

The next part of our script uses GridSearchCV to train and determine the best alpha parameter and saves the best score and the best alpha parameter in the best_score.txt file. The model is also saved in the my_model.pkl file. The code is illustrated in the following snippet:

with open('/pfs/out/best_score.txt', 'w', encoding='utf-8') as f:

      with redirect_stdout(f):

          for i, v in scoring.items():

             grid = GridSearchCV(estimator, params, cv=10, scoring= "r2")

             grid.fit(train_X, train_y)

             print(i)

             print('Best params:', grid.best_params_)

             if grid.best_score_ > 0:

                 print('Best score:', grid.best_score_)

             else:

                 print('Best score:', np.sqrt(abs(grid.best_score_)))

             print()

joblib.dump(estimator, '/pfs/out/my_model.pkl', compress =1)

Lastly, we plot our performance graph and save it to performance-plot.png.

Follow the next steps to create this pipeline:

Verify that Pachyderm is running by executing the following command:

pachctl version

You should see the following output. Note that your version of pachctl and pachd might vary:

COMPONENT           VERSION

pachctl             2.0.0

pachd               2.0.0

Create an evaluate pipeline by running the following command:

pachctl create pipeline -f evaluate.yaml

This command does not return any output.

View the active pipelines by running the following command:

pachctl list pipeline

You should see the following output:

NAME            VERSION INPUT         CREATED       STATE / LAST JOB   DESCRIPTION

evaluate        1       remove-outliers:/ 5 seconds ago  running / running A pipeline that evaluates the performance of the model.

train           1       remove-outliers:/ 34 seconds ago running / success A pipeline that trains the model with a selected estimator.

remove-outliers 1       data-clean:/* 5 seconds ago running / starting A pipeline that removes outliers from the dataset.

data-clean      1       data:/*       4 minutes ago running / success  A pipeline that removes empty cells from the CSV.

data-explore    1       data:/*       8 minutes ago running / success  A pipeline that performs exploratory analysis.

You should see the evaluate pipeline running or finished with a success status.

Look at the repository that the evaluate pipeline has created by running the following command:

pachctl list repo

You should see a new repository called train with 121KiB of data in it, as indicated in the following output:

NAME            CREATED            SIZE (MASTER) DESCRIPTION

evaluate        2 minutes ago  121KiB        Output repo for pipeline evaluate.

train           2 minutes ago 186.3KiB      Output repo for pipeline train.

remove-outliers About a minute ago 413.7KiB      Output repo for pipeline remove-outliers.

data-clean      49 minutes ago     780.4KiB      Output repo for pipeline data-clean.

data-explore    53 minutes ago     3.361MiB      Output repo for pipeline data-explore.

data            About an hour ago  449.9KiB

List the files in the evaluate repository by running the following command:

pachctl list file evaluate@master

The output should look like this:

NAME               TYPE SIZE

/best_score.txt       file 132B

/my_model.pkl         file 187B

/performance-plot.png file 120.7KiB

These files are our model, best MSE and R2 scores, and best alpha parameter, and a graph that shows how the training data compares to testing data.

Let's look at our best scores by running the following command:

pachctl get file evaluate@master:/best_score.txt

The output should look like this:

R_squared

Best params: {'alpha': 10}

Best score: 0.7040913319322766

MSE

Best params: {'alpha': 10}

Best score: 0.7040913319322766

Alpha 10 is our best alpha parameter. It should be used for predicting house prices.

The performance-plot.png file should show us how our training data evaluates against the testing data. We can view this by running the following command:

pachctl get file evaluate@master:/performance-plot.png | open -f -a "Preview.app"

Here is the graph it outputs:

Figure 9.13 – Performance plot

As you can see, alpha=10 is likely the best parameter in the range that we have provided. This pipeline concludes our example. The resulting model can be used to predict house prices with the trained alpha hyperparameter.

This concludes our example. Now, let's clean up our cluster.

Cleaning up

After you are done experimenting, you might want to clean up your cluster so that you start your next experiment with a fresh install. To clean up the environment, proceed as follows:

Delete all pipelines and repositories by running the following command:

pachctl delete pipeline –all && pachctl delete repo --all

Verify that no repositories and pipelines exist in your cluster by running the following command:

pachctl list repo && pachctl list pipeline

You should see the following output:

NAME CREATED SIZE (MASTER) DESCRIPTION

NAME VERSION INPUT CREATED STATE / LAST JOB DESCRIPTION

You have successfully cleaned up your cluster.

Summary

In this chapter, we have learned how to implement an ML pipeline that performs hyperparameter tuning on a house price prediction example. We've created five steps of this pipeline, each outputting relevant files and information into Pachyderm output repositories. In our first pipeline, we performed an exploratory analysis to gather a general understanding of the dataset and built a heatmap that helped us outline the correlation between various parameters in our dataset. In our second pipeline, we cleaned the data of columns with missing information, as well as removed parameters that have little influence on the sale price of a house. In our third pipeline, we removed outliers—values that were outside of the standard range. Our fourth pipeline split our dataset into two parts—one for testing and the other for training. And finally, our fifth pipeline performed hyperparameter tuning for the alpha parameter and found the best alpha for our use case. The last pipeline output our model in a .pkl file and created a graph where we could see the performance of our model against the training and testing dataset.

In the next chapter, we will learn about Pachyderm language clients. While you can do pure Python, R, or Scala in Pachyderm, you could also leverage one of our language clients or even build your own to take Pachyderm even further.

Further reading

Kaggle House Prices dataset: https://www.kaggle.com/lespin/house-prices-dataset

seaborn: https://seaborn.pydata.org/

Section 3:Pachyderm Clients and Tools

This section focuses on tools and libraries that Pachyderm can integrate with and provides practical examples of how to use Pachyderm language clients, as well as teaching you how to use Pachyderm Hub and Pachyderm Notebooks.

This section comprises the following chapters:

Chapter 10, Pachyderm Language Clients

Chapter 11, Using Pachyderm Notebooks

Chapter 10: Pachyderm Language Clients

In previous chapters, we have learned how to use Pachyderm through the pachctl Command-Line Interface (CLI). We have deployed and managed multiple Pachyderm pipelines by using mostly pachctl. We briefly looked into the Pachyderm User Interface (UI)—or the dashboard—as well, although we did not use it extensively. The Pachyderm CLI enables you to perform all Pachyderm management operations and, in general, provides more functionality than the Pachyderm UI.

However, many users may decide to take Pachyderm even further by using Pachyderm Application Programming Interfaces (APIs) directly without relying on pachctl or the dashboard. Many Pachyderm users develop scripts and call the Pachyderm API directly from these scripts. As of today, Pachyderm provides two official Pachyderm programming language clients, Golang (Go) and Python, to enable advanced users to extend Pachyderm functionality further.

In addition, if you are familiar with Protocol Buffers (Protobuf), an open source platform that enables cross-platform development, you can leverage the Pachyderm pps.proto file to access Pachyderm through such languages as C, C++, and Java.

In this chapter, you will learn how to use both Python and Go Pachyderm clients. You will learn how to run basic operations by using both of these clients, including how to create repositories and pipelines.

This chapter is intended to demonstrate how to use official Pachyderm language clients.

We will cover the following topics:

Using the Pachyderm Go client

Cloning the Pachyderm source repository

Using the Pachyderm Python client

Technical requirements

You should have already installed the components listed next.

For a local macOS installation, you will need the following:

macOS Mojave, Catalina, Big Sur, or later

Docker Desktop for Mac 10.14

minikube v1.9.0 or later

pachctl 2.0.0 or later

Pachyderm 2.0.0 or later

For a local Windows installation, you will need the following:

Windows Pro 64-bit v10 or later

Windows Subsystem for Linux (WSL) 2 or later

Microsoft PowerShell v6.2.1 or later

Hyper-V

minikube v1.9.0 or later

kubectl v1.18 or later

pachctl 2.0.0 or later

Pachyderm 2.0.0 or later

For an Amazon Elastic Kubernetes Service (Amazon EKS) installation, you will need the following:

kubectl v1.18 or later

eksctl

aws-iam-authenticator

pachctl 2.0.0 or later

Pachyderm 2.0.0 or later

For a Microsoft Azure Kubernetes Service (AKS) cloud installation, you will need the following:

kubectl v1.18 or later

The Azure CLI

pachctl 2.0.0 or later

Pachyderm 2.0.0 or later

jq 1.5 or later

For a Google Kubernetes Engine (GKE) cloud installation, you will need the following:

Google Cloud Software Development kit (SDK) v124.0.0 or later

kubectl v1.18 or later

pachctl 2.0.0 or later

Pachyderm 2.0.0 or later

Downloading the source files

All scripts for this chapter are available at https://github.com/PacktPublishing/Reproducible-Data-Science-with-Pachyderm/tree/main/Chapter10-Pachyderm-Language-Clients.

We will use the image processing example that we had in Chapter 6, Creating Your First Pipeline. If you do not have them already, download the files for this example from here: https://github.com/PacktPublishing/Reproducible-Data-Science-with-Pachyderm/tree/main/Chapter06-Creating-Your-First-Pipeline.

Using the Pachyderm Go client

The Pachyderm Go client enables you to communicate with Pachyderm through the Go API. Go is a popular programming language that has been developed by Google and has become widely popular among the developer community in recent years. In this chapter, we will learn how to enable the Pachyderm Go client and how to run basic Pachyderm operations using the Go client.

The main source files that you can use for reference are located in the https://github.com/pachyderm/pachyderm/tree/master/src/client directory of the Pachyderm source repository. These files include all the methods that you can use to communicate with Pachyderm objects and primitives—specifically, the following files:

https://github.com/pachyderm/pachyderm/blob/master/src/client/client.go

https://github.com/pachyderm/pachyderm/blob/master/src/client/pfs.go

https://github.com/pachyderm/pachyderm/blob/master/src/client/pps.go

These files include most of the important Pachyderm methods and the ones that we will use in the following sections.

Installing Go on your computer

To get started, we need to verify that we have a valid Go installation in our environment. Go is supported in all major operating systems, including Microsoft Windows, Linux, and macOS. Check that you have Go installed on your computer by running the following command:

go version

You should see an output similar to this:

go version go1.16.4 darwin/amd64

If you do not have Go installed on your machine, follow these next steps to install it:

Go to https://golang.org/doc/install and download the version of Go for your operating system.

Open the downloaded package and follow the prompts to install Go in your system. You should see the following screen when you are done:

Figure 10.1 – Go installation

Follow the instructions for your operating system to verify that Go was installed as described in https://golang.org/doc/install.

Restart your Terminal and run go version again to verify your installation. You should see an output similar to this:

go version go1.16.4 darwin/amd64

Now that we have Go installed, let's configure $GOPATH.

Configuring $GOPATH

If you have never used Go before, you need to make sure that you have your $GOPATH directory properly set up; otherwise, none of the scripts described in this section will work. When you installed Go, it might have already been configured. However, you might still want to configure the following:

Verify that you have the following in either ~/.bash_profile, ~/.profile, or ~/.zsh_profile file:

export GOPATH="$HOME/go"

PATH="$GOPATH/bin:$PATH"

If your respective shell configuration file did not have this configuration, add it, and then source your shell configuration file, like this:

source ~/.<shell-profile>

Check your $GOPATH directory by running the following command:

go env

This command prints your Go environment configuration. If you are on macOS, your $GOPATH directory should be `GOPATH="/Users/<username>/go"`.

If you don't have it already, create a src directory under your $GOPATH directory, as follows:

mkdir $GOPATH/src

Under $GOPATH/src, create a github.com directory, as follows:

mkdir $GOPATH/src/github.com

You will need to clone the Pachyderm repository into this directory, as described in the next chapter.

Update to the latest version of grpc, as follows:

go get google.golang.org/grpc

After you have configured $GOPATH, you need to clone the Pachyderm source repository.

Cloning the Pachyderm source repository

Before you can use Pachyderm language clients, you need to have a copy of the Pachyderm source repository on your machine to be able to use the APIs. You will run the client methods against an existing Pachyderm cluster. The Pachyderm repository is stored in GitHub at https://github.com/pachyderm/pachyderm. In addition, you need to make sure that you switch to the branch and tag that matches your pachd and pachctl version. In this section, we will learn how to clone the Pachyderm repository and how to switch to the required branch and tag.

To be able to run Go modules used in the scripts in this section, you need to clone the Pachyderm repository under the $GOPATH directory on your computer. On Mac, Go is installed under /Users/<username>/go, and you can clone the Pachyderm repository at / Users/<username>/go/src/github.com/.

To clone the Pachyderm repository, complete the following steps:

Go to https://github.com/pachyderm/pachyderm.

Click on the Code tab, as shown in the following screenshot:

Figure 10.2 – Pachyderm source repository

In the drop-down menu, select an option to clone with HTTPS or SSH, and click the Clone icon.

Important note

If you decide to clone with Secure Shell (SSH) and this is your first time cloning from GitHub, you will likely need to configure an SSH key pair. For more information, see https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh.

Go to your terminal and run the git clone command with the HyperText Transfer Protocol Secure (HTTPS) or SSH address you copied in Step 3, as follows:

git clone git@github.com:pachyderm/pachyderm.git

You should see an output similar to this:

Cloning into 'pachyderm'...

remote: Enumerating objects: 226153, done.

remote: Counting objects: 100% (171/171), done.

...

The Pachyderm source code will be cloned to the pachyderm directory.

Go to the pachyderm directory by running the following command:

cd pachyderm

Check the branch you are on by running the following command:

git branch

Get a list of tags by running the following command:

git fetch –tags

Verify the version of pachctl and pachd that you are using by running the following command:

pachctl version

Your output might look like this:

COMPONENT           VERSION

pachctl             2.0.1

pachd               2.0.1

Check out the tag that corresponds to the version of pachctl and pachd that you are using. In this example, we need to check out the 2.0.1 tag:

git checkout tags/v2.0.1

Check that you have switched to the correct version by running the following command:

git branch

You should see the following output:

* (HEAD detached at v2.0.1)

  master

We now have valid Pachyderm source code that we will use to access our Pachyderm cluster. Next, let's connect to Pachyderm with the Go client.

Connecting to Pachyderm with the Go client

You must have a running Pachyderm cluster to be able to use the Go API client. If you have followed previous sections, you likely have one running on a cloud platform of your choice or locally. If not, go back to Chapter 4, Installing Pachyderm Locally, or Chapter 5, Installing Pachyderm on a Cloud Platform, and deploy a cluster.

We will use the access.go script to access Pachyderm. Let's look at the script to understand how it works. The first part of the script imports the required components, as we can see here:

package main

import (

     "fmt"

     "log"

     "github.com/gogo/protobuf/types"

     "github.com/pachyderm/pachyderm/v2/src/client"

)

The second part of the script defines a main function. You must use the main function with Go; otherwise, it won't work. The main function defines the Internet Protocol (IP) address and port of your cluster. In the following code example, it is localhost or 127.0.0.1. 30650 is the pachd port:

func main() {

      c, err := client.NewFromURI("grpc://localhost:30650")

      if err != nil {

          log.Fatal(err)

      }

The third part of the script, shown here, gets the version of your cluster:

     version, err := c.VersionAPIClient.GetVersion(c.Ctx(), &types.Empty{})

     if err != nil {

         panic(err)

     }

     fmt.Println(version)

}

To connect to a Pachyderm cluster, you need to know the IP address of your cluster. If running these examples on a local machine, grpc://localhost:30650 should work.

Now, let's run the script. Follow these next steps:

Unless you have a load balancer deployed that enables access to your cluster, you also need to make sure that you run Pachyderm port-forwarding all the time you are accessing your cluster through the API. To start Pachyderm port-forwarding, run the following command in a separate terminal window:

pachctl port-forward

Run the access.go script, as follows:

go run access.go

This is an example response that you should get:

major:2 micro:1

We have successfully accessed our cluster through the Go API. Our cluster is running version 2.0.1. Your version might be different.

Now, let's use the Go API to create a Pachyderm repository.

Creating a repository with the Go client

Now that we know how to connect to Pachyderm, let's create a repository with the code we have in the create-repo.go script.

Here is what the script imports:

package main

import (

     "fmt"

     "log"

     "github.com/pachyderm/pachyderm/v2/src/client"

     "github.com/pachyderm/pachyderm/v2/src/pfs"

)

The next part of the script defines a main function that performs the following operations:

Connects to the Pachyderm cluster.

Creates a repository called photos.

Lists the repositories on this cluster.

Here is how it looks:

func main() {

     c, err := client.NewOnUserMachine("user")

     if err != nil {

         log.Fatal(err)

     }

     if _, err := c.PfsAPIClient.CreateRepo(

         c.Ctx(),

         &pfs.CreateRepoRequest{

             Repo:        client.NewRepo("photos"),

             Description: "A repository that stores images.",

             Update:      true,

         },

     ); err != nil {

         panic(err)

     }

     repos, err := c.ListRepo()

     if err != nil {

         log.Fatal(err)

     }

     fmt.Println(repos)

}

You must run port-forwarding and make sure you replace the IP address listed in the script with the IP address of your cluster. If you are running the cluster in minikube, you probably don't need to change anything.

Run the create-repo.go script, as follows:

go run create-repo.go

This command returns the following output:

[repo:<name:"photos" type:"user" > created:<seconds:1637264349 nanos:440180000 > description:"A repository that stores images." auth_info:<permissions:REPO_READ permissions:REPO_INSPECT_COMMIT permissions:REPO_LIST_COMMIT permissions:REPO_LIST_BRANCH permissions:REPO_LIST_FILE permissions:REPO_INSPECT_FILE permissions:REPO_ADD_PIPELINE_READER permissions:REPO_REMOVE_PIPELINE_READER permissions:PIPELINE_LIST_JOB permissions:REPO_WRITE permissions:REPO_DELETE_COMMIT permissions:REPO_CREATE_BRANCH permissions:REPO_DELETE_BRANCH permissions:REPO_ADD_PIPELINE_WRITER permissions:REPO_MODIFY_BINDINGS permissions:REPO_DELETE roles:"repoOwner" > ]

Now that we have created a repository, let's put some data into it.

Putting data into a Pachyderm repository with the Go client

In the previous section, we created a Pachyderm repository called photos. Let's add the files that we have used in Chapter 6, Creating Your First Pipeline, to this repository. We will use the put-files.go script to add the files. Here is what the script imports:

package main

import (

     "fmt"

     "log"

     "os"

     "github.com/pachyderm/pachyderm/v2/src/client"

)

The next part of the script connects to the Pachyderm cluster and adds the landscape.png, red_vase.png, and hand.png files to the master branch of the photos repository.

Here is the part that connects to the repository. Make sure you replace the IP address with the address of your cluster:

func main() {

     c, err := client.NewOnUserMachine("user")

     if err != nil {

         log.Fatal(err)

     }

This part adds the files:

     myCommit := client.NewCommit("photos","master", "")

     f1, err := os.Open("landscape.png")

     if err != nil {

         panic(err)

     }

     if err := c.PutFile(myCommit, "landscape.png", f1); err != nil {

         panic(err)}

     f2, err := os.Open("brown_vase.png")

     if err != nil {

         panic(err)

     }

     if err := c.PutFile(myCommit, "brown_vase.png", f2); err != nil {

         panic(err)

     }

     f3, err := os.Open("hand.png")

     if err != nil {

         panic(err)

     }

     if err := c.PutFile(myCommit, "hand.png", f3); err != nil {

         panic(err)

     }

And the last part, shown here, lists the files in the master branch of the photos repository:

    files, err := c.ListFileAll(myCommit, "/")

    if err != nil {

        panic(err)

    }

    fmt.Println(files)

}

Let's run this script with the following command:

go run put-files.go

This script returns the following output:

[file:<commit:<branch:<repo:<name:"photos" type:"user" > name:"master" > id:"2c15226b838f48cabd2ae13b43c26517" > path:"/brown_vase.png" datum:"default" > file_type:FILE committed:<seconds:1637299733 nanos:503150000 > size_bytes:93481 hash:"\2061\023\026\376O&\323\313\212\215\226Ra\346\245=Er\r_@E\023\360\352\240\275}\204\235\346"  file:<commit:<branch:<repo:<name:"photos" type:"user" > name:"master" > id:"2c15226b838f48cabd2ae13b43c26517" > path:"/hand.png" datum:"default" > file_type:FILE committed:<seconds:1637299733 nanos:503150000 > size_bytes:856063 hash:"\014X\224\032\0251\260(\263\267\234\345{\016\353a\0310\3579\354\323\372\013\357yFg\274\256\000}"  file:<commit:<branch:<repo:<name:"photos" type:"user" > name:"master" > id:"2c15226b838f48cabd2ae13b43c26517" > path:"/landscape.png" datum:"default" > file_type:FILE committed:<seconds:1637299733 nanos:503150000 > size_bytes:54009 hash:"\320:\265\036\3363z&\264\324]\364unfv\243\300\001[\206\347\344b\257\274\366\220JnR\004" ]

Great! We have a repository with data in it. Now, let's learn how to create a pipeline.

Creating pipelines with the Go client

Finally, we can create pipelines for our example from Chapter 6, Creating Your First Pipeline.

Here is what the create-pipeline.go script imports:

package main

import (

     "fmt"

     "log"

     "github.com/pachyderm/pachyderm/v2/src/client"

     "github.com/pachyderm/pachyderm/v2/src/pps""

)

The second part of the script connects to the Pachyderm cluster by using the pachd IP address, as follows:

func main() {

     c, err := client.NewFromAddress("127.0.0.1:30650")

     if err != nil {

         log.Fatal(err)

     }

The next part of the script creates a contour pipeline. You can see that the script uses the svekars/contour-histogram:1.0 image and gets the data from the photos repository with the / glob pattern. One important thing to note in the example shown here is that you need to specify the parallelism_spec for all pipelines:

if err := c.CreatePipeline(

         "contour",

         "svekars/contour-histogram:1.0 ",

         []string{"python3", "/contour.py"},

         []string{},

         &pps.ParallelismSpec{

             Constant: 1,

         },

         client.NewPFSInput("photos", "/"),

         "",

         false,

     ); err != nil {

         panic(err)

     }

Next, the script creates a histogram pipeline, as follows:

if err := c.CreatePipeline(

         "histogram",

         "svekars/contour-histogram:1.0",

         []string{"python3", "/histogram.py"},

         []string{},

         &pps.ParallelismSpec{

             Constant: 1,

         },

         client.NewPFSInput("contour", "/"),

         "",

         false,

     ); err != nil {

         panic(err)

     }

And finally, the script lists all the created pipelines, as follows:

     pipelines, err := c.ListPipeline(true)

     if err != nil {

         panic(err)

     }

     fmt.Println(pipelines)

}

Run the following command:

go run create-pipeline.go

Here is an example response:

[pipeline:<name:"histogram" > version:1 spec_commit:<branch:<repo:<name:"histogram" type:"spec" > name:"master" > id:"44945b0d0e2944e3b1015617e224e3e3" > state:PIPELINE_STARTING job_counts:<key:1 value:1 > last_job_state:JOB_CREATED parallelism:1 type:PIPELINE_TYPE_TRANSFORM details:<transform:<image:"svekars/contour-histogram:1.0" cmd:"python3" cmd:"/histogram.py" > parallelism_spec:<constant:1 > created_at:<seconds:1637300756 nanos:806783300 > output_branch:"master" input:<pfs:<name:"contour" repo:"contour" repo_type:"user" branch:"master" glob:"/" > > salt:"0715a02027ba4489a79bd8a400f349ad" datum_tries:3 reprocess_spec:"until_success" >  

pipeline:<name:"contour" > version:1 spec_commit:<branch:<repo:<name:"contour" type:"spec" > name:"master" > id:"f3f8bf226e5a4dda8a9f27da10b7fd87" > state:PIPELINE_STARTING job_counts:<key:1 value:1 > last_job_state:JOB_CREATED parallelism:1 type:PIPELINE_TYPE_TRANSFORM details:<transform:<image:"svekars/contour-histogram:1.0 " cmd:"python3" cmd:"/contour.py" > parallelism_spec:<constant:1 > created_at:<seconds:1637300756 nanos:592992600 > output_branch:"master" input:<pfs:<name:"photos" repo:"photos" repo_type:"user" branch:"master" glob:"/" > > salt:"98c0a867ea56439eb1f2466fbf1aa838" datum_tries:3 reprocess_spec:"until_success" > ]

You can see that the script has created two of our pipelines, as intended. We have the whole example uploaded in one file called contour-go-example.go in the chapter's GitHub repository. Now that you have learned how to do it, you can just run that one script to create a whole contour pipeline example from one command. Next, we'll learn how to clean up our cluster.

Cleaning up the cluster with the Go client

The cleanup.go script cleans up your cluster and deletes all the pipelines, data, and repositories. Only run it if you do not want to preserve the data anymore.

This script only needs to import the client from the Pachyderm repository. For this, the following code is required:

package main

import (

     "fmt"

     "log"

     "github.com/pachyderm/pachyderm/v2/src/client"

)

The next part of the script deletes all the repositories and pipelines. We set the force flag to true for all pipelines and repositories so that Pachyderm does not interrupt deletion due to downstream pipeline dependencies. The code is illustrated in the following snippet:

     if err := c.DeleteRepo("contour", true); err != nil {

         panic(err)

     }

     if err := c.DeleteRepo("photos", true); err != nil {

          panic(err)

     }

     if err := c.DeleteRepo("histogram", true); err != nil {

           panic(err)

     }

     if err := c.DeletePipeline("contour", true); err != nil {

           panic(err)

     }

     if err := c.DeletePipeline("histogram", true); err != nil {

           panic(err)

     }

And the final part of the script returns empty lists since we delete all the pipelines and repositories, as illustrated in the following code snippet:

pipelines, err := c.ListPipeline(true)

     if err != nil {

         panic(err)

     }

     fmt.Println(pipelines)

     repos, err := c.ListRepo()

     if err != nil {

         log.Fatal(err)

     }

     fmt.Println(repos)

}

Run the following command:

go run cleanup.go

This command returns the following output:

[]

[]

In this section, we have learned how to use the Go client to create Pachyderm pipelines and repositories. Next, let's learn how to do this with the Pachyderm Python client.

Using the Pachyderm Python client

Python is probably one of the most popular languages within the software engineering and data science community. Pachyderm provides an officially supported Python client through the python-pachyderm package. You can find the Python Pachyderm source repository on GitHub at https://github.com/pachyderm/python-pachyderm and on the Python Package Index (PyPI) at https://pypi.org/project/python-pachyderm/.

The main files that you can use as a Python client reference are located in the https://github.com/pachyderm/python-pachyderm/tree/master/src/python_pachyderm/mixin directory of the Pachyderm source repository—most notably, the following files:

https://github.com/pachyderm/python-pachyderm/blob/master/src/python_pachyderm/client.py

https://github.com/pachyderm/python-pachyderm/blob/master/src/python_pachyderm/mixin/pfs.py

https://github.com/pachyderm/python-pachyderm/blob/master/src/python_pachyderm/mixin/pps.py

Before you proceed, you must have the following components configured on your machine:

A copy of the Pachyderm repository (see the Cloning the Pachyderm source repository section). With Python Pachyderm, you can clone your repository to any directory on your machine. It does not have to be $GOPATH.

Python 3.6 or later installed on your machine.

Access to an active Pachyderm cluster. If it's a local installation, you need to have Pachyderm port-forwarding running all the time you work with the repository through the APIs. If it's a cloud installation, you need to either have a load balancer running to enable access to your cluster, or you might be able to use Pachyderm port-forwarding as well.

We have reviewed the prerequisites for this section. Now, let's install the python-pachyderm client.

Installing the Pachyderm Python client

Before you can start using the Pachyderm Python client, you need to install it on your machine.

To install the Python Pachyderm client, complete the following steps:

Open a terminal window.

If you are on macOS or Linux, run the following command:

pip install python-pachyderm

  You should see the following output:

Collecting python-pachyderm

  Downloading python-pachyderm-6.2.0.tar.gz (409 kB)

  ...

Successfully installed grpcio-1.38.0 protobuf-3.17.0 python-pachyderm-6.2.0

Your version of the python-pachyderm package might be different.

Now that we have python-pachyderm installed, let's connect to Pachyderm by using python-pachyderm.

Connecting to your Pachyderm cluster with the Python client

To get started, let's use the access.py script to connect to your cluster. Make sure port-forwarding is running on your machine. Here is the script:

import python_pachyderm

client = python_pachyderm.Client()

print(client.get_remote_version())

This script connects to pachd, which runs on https://localhost:30650, by using the python_pachyderm.Client() invocation and prints the version of Pachyderm that you are running.

Let's run this script and see what output it returns.

Run the access.py script with the following command:

python access.py

You should see output similar to this:

major: 2

micro: 1

This output means that we are on version 2.0.1. Your output might be different.

Now that we know how to access the cluster, let's go ahead and create a Pachyderm repository.

Creating a Pachyderm repository with the Python client

We will use the create-repo.py script to create a Pachyderm repository called photos.

Here is the code of the script:

import python_pachyderm

client = python_pachyderm.Client()

client.create_repo("photos")

print(list(client.list_repo()))

Run the create-repo.py script with the following command:

python create-repo.py

Here is an example output:

[repo {

  name: "photos"

  type: "user"

}

created {

  seconds: 1637207890

  nanos: 80987000

}

auth_info {

  permissions: REPO_READ

  permissions: REPO_INSPECT_COMMIT

...

Now that we have a repository created, let's put some data into it.

Putting data into a Pachyderm repository with the Python client

We will put the same files we used in Chapter 6, Creating Your First Pipeline, to the photos repository we have just created. Here is the script that we will use:

import python_pachyderm

client = python_pachyderm.Client()

with client.commit('photos', 'master') as i:

     client.put_file_url(i, 'landscape.png', 'https://i.imgur.com/zKo9Mdl.jpg')

     client.put_file_url(i, 'hand.png', 'https://i.imgur.com/HtZ8FyG.png')

     client.put_file_url(i, 'red_vase.png', 'https://i.imgur.com/d45jop9.jpg') print(list(client.list_file(("photos","master"), "/")))

The script uses the client.commit method to start a commit to the master branch of the photos repository, and the client.put_file_bytes adds three files to the repository. Note that client.list_file needs to be a list and not a string for the command to work correctly.

Let's run this script.

Run the put-files.py script with the following command:

python put-files.py

Here is the system response that you should get:

[file {

  commit {

    branch {

      repo {

        name: "photos"

        type: "user"

      }

      name: "master"

    }

    id: "e29c6f5c49244ce193fe5f86c9df0297"

  }

  path: "/hand.png"

  datum: "default"

}

file_type: FILE

committed {

  seconds: 1637208291

  nanos: 161527000

}

size_bytes: 856063

hash: "\014X\224\032\0251\260(\263\267\234\345{\016\353a\0310\3579\354\323\372\013\357yFg\274\256\000}"  

...

]

The preceding output is truncated. You should see the same output for each file that we added to the repository.

Now that we have added the files, let's create pipelines for this example.

Creating pipelines with the Pachyderm Python client

Now that we have a repository and files uploaded to it, let's use the create-pipeline.py script to create two pipelines from the example we had in Chapter 6, Creating Your First Pipeline.

python-pachyderm provides two methods to create pipelines, as outlined here:

create_pipeline: This method is for all languages and is equivalent to the pachctl create pipeline method.

create_python_pipeline: This pipeline is designed to be run with Python code and provides a slightly different User Experience (UX). You can read more about this method in the Pachyderm documentation, at https://docs.pachyderm.com.

We will use the standard create_pipeline method to create this pipeline.

The first part of the script creates a contour pipeline, as follows:

import python_pachyderm

from python_pachyderm.service import pps_proto

client = python_pachyderm.Client()

client.create_pipeline(

     pipeline_name="contour",

     transform=pps_proto.Transform(

         cmd=["python3", "contour.py"],

         image="svekars/contour-histogram:1.0",

     ),

     input=pps_proto.Input(

         pfs=pps_proto.PFSInput(glob="/", repo="photos")

     ),

)

The second part of the script creates a histogram pipeline, as follows:

client.create_pipeline(

     pipeline_name="histogram",

     transform=pps_proto.Transform(

         cmd=["python3", "histogram.py"],

         image="svekars/contour-histogram:1.0",

     ),

     input=pps_proto.Input(

         pfs=pps_proto.PFSInput(glob="/", repo="contour")

     ),

)

And the last part of the script returns a list of pipelines that were created, as illustrated in the following code snippet:

Print(list(client.list_pipeline()))

Let's run this script.

Run the create-pipeline.py script with the following command:

python create-pipeline.py

Here is a fragment of the output:

[pipeline {

  name: "histogram"

}

version: 1

spec_commit {

  branch {

    repo {

      name: "histogram"

      type: "spec"

    }

    name: "master"

  }

  id: "94286ef36318425c8177bd4e0f959c57"

}

state: PIPELINE_STARTING

job_counts {

  key: 1

  value: 1

}...

In this section, we have learned how to create pipelines by using the python-pachyderm client. Next, let's clean up our cluster.

Cleaning up the cluster with the Python client

We have successfully recreated our contour and histogram pipeline example. The whole example is available as one file called contour-histogram-example.py in the GitHub repository. You can download it at https://github.com/PacktPublishing/Reproducible-Data-Science-with-Pachyderm/tree/main/Chapter10-Pachyderm-Language-Clients and recreate it as many times as needed.

In this section, we will clean up our cluster so that we have a clean installation for Chapter 11, Using Pachyderm Notebooks. We will use the cleanup.py script for that, as follows:

import python_pachyderm

client.delete_repo("photos", force=True)

client.delete_pipeline(pipeline_name="contour", force=True, keep_repo=False)

client.delete_pipeline(pipeline_name="histogram", force=True, keep_repo=False)

print(list(client.list_repo()))

print(list(client.list_pipeline()))

This script uses the delete_all_pipelines method, which deletes all pipelines in the cluster. You can also use delete_all to delete all objects and primitives in the cluster.

Let's run this script.

Run the cleanup.py script with the following command:

python cleanup.py

This command should return the following output:

[]

[]

That's it! We have successfully cleaned up our cluster.

Summary

In this chapter, we've learned about how to use the two officially supported Pachyderm language clients—the Pachyderm Go client and the Python client. We've learned how to clone the Pachyderm repository and switch to the correct branch and tag. We've learned how to connect, create repositories, put files into repositories, and create pipelines, as well as delete all the objects after we are done. There is much more that you can do with these two language clients, but the examples in this chapter give you a general idea about how to use them.

In the next chapter, we will learn how to integrate Pachyderm with JupyterHub, a popular data science Integrated Development Environment (IDE) for which Pachyderm has a special plugin. We will also work more with the python-pachyderm client.

Further reading

Setting up Ingress with Traefik to access the Pachyderm UI: https://docs.pachyderm.com/latest/deploy-manage/deploy/ingress/pach-ui-ingress/#traefik-ingress-controller-on-pachyderm-uis-cluster-in-one-diagram

Connecting to GitHub with SSH: https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh

Go documentation: https://golang.org/doc/

Python 3 documentation: https://docs.python.org/3/

Python Pachyderm client documentation: https://python-pachyderm.readthedocs.io/en/stable/

Chapter 11: Using Pachyderm Notebooks

In Chapter 10, Pachyderm Language Clients, we learned how to use Pachyderm language clients, including the Pachyderm Go client and Pachyderm Python client. The latter is probably more popular among data scientists as Python is the language that a lot of data scientists use. And if you write in Python, you are likely familiar with the open source tool, JupyterLab.

JupyterLab is an open source platform that provides an Interactive Development Environment (IDE) in which you can not only author your code but also execute it. This advantage makes JupyterLab an ideal tool for data science experiments. However, while JupyterLab provides a basic version control system for its notebooks, it's not to the level of data provenance that Pachyderm offers. Pachyderm Hub, the SaaS version of Pachyderm, provides a way to integrate your Pachyderm cluster with Pachyderm Notebooks, a built-in version of JupyterLab coupled with Pachyderm.

This chapter is intended to demonstrate how to configure a Pachyderm cluster in Pachyderm Hub and use Pachyderm Notebooks. By the end of this chapter, we will learn how to run basic Pachyderm operations in Pachyderm Notebooks and will create a sentiment analysis pipeline.

This chapter covers the following topics:

Enabling Pachyderm Notebooks in Pachyderm Hub

Running basic Pachyderm operations in Pachyderm Notebooks

Creating and running an example pipeline in Pachyderm Notebooks

Technical requirements

You should have already installed the following components:

pachctl 2.0.0 or later

Access to Pachyderm Hub

A GitHub or Gmail account

Downloading the source files

All code samples used in this section are stored in the GitHub repository created for this book at https://github.com/PacktPublishing/Reproducible-Data-Science-with-Pachyderm/tree/main/Chapter11-Using-Pachyderm-Notebooks.

The Dockerfile used in this section is stored at https://hub.docker.com/repository/docker/svekars/pachyderm-ide.

Enabling Pachyderm Notebooks in Pachyderm Hub

Before you can take advantage of Pachyderm Notebooks, you need to create an account in Pachyderm Hub and a Pachyderm workspace. Pachyderm Hub provides a trial period for all users to test its functionality.

After the trial period ends, you need to upgrade to the Pachyderm Pro version to continue using Pachyderm Notebooks.

Create a workspace

In Pachyderm Hub, your work is organized in workspaces. A workspace is a grouping in which multiple Pachyderm clusters can run. Your organization might decide to assign each workspace to a team of engineers.

Before you can create a workspace, you need a Pachyderm Hub account, so let's create one. Pachyderm Hub supports authentication with Gmail and GitHub. You must have either of those to create an account with Pachyderm Hub.

To create a Pachyderm Hub account, complete the following steps:

Go to https://www.pachyderm.com/try-pachyderm-hub# and fill out the provided form to request a free trial of Pachyderm Hub. It might take some time for your account to be activated and you will receive an email with instructions.

Log in to your Pachyderm Hub account with the Gmail or GitHub account you have provided in the sign-up form.

Fill out the Get Started form and click Get Started.

Click Create a workspace, fill out the form that appears as shown in the following screenshot, and click Create:

Figure 11.1 – Create a new workspace

When you create a workspace, you automatically deploy a Pachyderm cluster. If you're using a trial version of Pachyderm, you will have deployed a single-node cluster, which should be enough for testing.

Now that you have created your first workspace, you need to connect to your cluster using pachctl.

Connect to your Pachyderm Hub workspace with pachctl

If this is not the first chapter that you are reading in this book, you should already have pacthl installed on your computer. Otherwise, install pachctl as described in Chapter 4, Installing Pachyderm Locally.

To connect to your Pachyderm Hub workspace, do the following:

In the Pachyderm Hub UI, locate your workspace, and click the CLI link.

Follow the instructions in the UI to switch your Pachyderm context and enable communication between your computer and your workspace on Pachyderm Hub.

After you authenticate, run the following command:

pachctl version

You should get the following response:

COMPONENT           VERSION

pachctl             2.0.1

pachd               2.0.1

Check that you have switched to the correct context by running the following command:

pachctl config get active-context

This command should return the name of your Pachyderm Hub workspace.

You can now communicate with your cluster deployed on Pachyderm Hub through pachctl from a terminal on your computer.

Now that we have configured our cluster, let's connect to a Pachyderm notebook.

Connect to a Pachyderm notebook

Pachyderm Notebooks is an IDE for data scientists that provides easy access to familiar Python libraries. You can run and test your code in cells while Pachyderm backs your pipeline.

To connect to a Pachyderm notebook, complete the following steps:

In the Pachyderm Hub UI, in your workspace, click Notebooks:

Figure 11.2 – Access Pachyderm Notebooks

When prompted, click Sign in with OAuth 2.0 and then sign in with the account that has access to Pachyderm Hub.

On the Server Options screen, select the Default server:

Figure 11.3 – Server Options

Click Start and then Launch Server.

You should see the following screen:

Figure 11.4 – Pachyderm Notebooks home

Now that we have access to Pachyderm Notebooks, we can create Pachyderm pipelines directly from the Pachyderm Notebooks UI, experiment with Python code and python-pachyderm, run pachctl commands, and even create Markdown files to document our experiments. We'll look into this functionality in the next section.

Running basic Pachyderm operations in Pachyderm Notebooks

The main advantage of Pachyderm Notebooks is that it provides a unified experience. Not only can you run your experiments inside of it, but you can also access your Pachyderm cluster through the integrated terminal by using both pachctl and python-pachyderm. All the pipelines that you create through the JupyterLab UI will be reflected in your Pachyderm cluster whether it runs locally or on a cloud platform.

Now let's see how we can use Pachyderm Notebooks to access our Pachyderm cluster.

Using the integrated terminal

You can run the integrated terminal from within Pachyderm Notebooks and use it to execute pachctl or any other UNIX commands.

To use the integrated terminal, complete the following steps:

On the Pachyderm Notebooks home page, click Terminal.

Click the Terminal icon to start a new Terminal window:

Figure 11.5 – Start a Terminal within Pachyderm Notebooks

Get the version of Pachyderm:

pachctl version

Try running other Pachyderm commands that you have learned in previous chapters to see how it works.

Important note

The version of pachctl and pachd are different than the ones your run directly from your computer terminal as Pachyderm Notebooks has a preinstalled version of pachctl, which sometimes might not match the version of your cluster. This should not affect your work with Pachyderm.

Now that we know how to use the terminal, let's try to create a Pachyderm notebook.

Using Pachyderm Notebooks

A notebook is an interactive document in which the users can write Python code, run it, and visualize it. These features make notebooks a great experimentation tool that many data scientists use in their work. After the experiment is finished, you might want to export the notebook as a Python script or library.

Pachyderm Notebooks supports the following types of notebooks:

Python notebooks

Julia notebooks

R notebooks

These three languages seem to be most popular among data scientists. You can create Julia and R notebooks to experiment specifically with the code that you want to use with your pipeline. With Python notebooks, not only you can test your code, but you can also use the python-pachyderm client to interact with the Pachyderm cluster.

Important note

The code described in this section can be found in the https://github.com/PacktPublishing/Reproducible-Data-Science-with-Pachyderm/blob/main/Chapter11-Using-Pachyderm-Notebooks/example.ipynb file.

Let's create a Python notebook and run a few commands:

On the Pachyderm Notebooks home screen, click on the Python 3 notebook icon to create a new notebook.

We can use both regular Python and python-pachyderm in this notebook. For example, to get the current version of your Pachyderm cluster and list all the repositories, paste the following in the notebook's cell:

import python_pachyderm

client = python_pachyderm.Client()

print(client.get_remote_version())

print(list(client.list_repo()))

Click the run icon to run the script and get the result:

Figure 11.6 – Running python-pachyderm in Notebooks

Create a repository by pasting the following code in the next cell and running it:

client.create_repo("data")

print(list(client.list_repo()))

Here is the output that you should see:

Figure 11.7 – List repo

Note that you do not need to import python_pachyderm and define the client in the second and subsequent cells since you have already defined it in the first cell.

Let's add some files to the data repository:

with client.commit('data', 'master') as i:

    client.put_file_url(i, 'total_vaccinations_dec_2020_24-31.csv', 'https://raw.githubusercontent.com/PacktPublishing/Reproducible-Data-Science-with-Pachyderm/main/Chapter11-Using-Pachyderm-Notebooks/total_vaccinations_dec_2020_24-31.csv')

print(list(client.list_file(("data", "master"), "")))

Here is the output that you should see:

Figure 11.8 – Put file output

This dataset contains statistics about COVID-19 vaccinations from December 24 to 31, 2020.

You can print the contents of the file by running the following code:

import pandas as pd

pd.read_csv(client.get_file(("data", "master"), "total_vaccinations_dec_2020_24-31.csv"))

You should see the following output:

Figure 11.9 – Total vaccinations

This CSV file has the following columns: location, date, vaccine (manufacturer), and total_vaccinations.

Let's also print the top five rows of the file to view the names of the columns:

import pandas as pd

df = pd.read_csv(client.get_file(("data", "master"), "total_vaccinations_dec_2020_24-31.csv"))

data_top = df.head()

print(data_top)

You should see the following output:

Figure 11.10 – Top five rows

Finally, let's create a simple pipeline that will tell us which country did the most vaccinations in a single given day during the observed period:

from python_pachyderm.service import pps_proto

client.create_pipeline(

     pipeline_name="find-vaccinations",

     transform=pps_proto.Transform(

         cmd=["python3"],

         stdin=[

             "import pandas as pd",

             "df = pd.read_csv('/pfs/data/total_vaccinations_dec_2020_24-31.csv')",

             "max_vac = df['total_vaccinations'].idxmax()",

             "row = df.iloc[[max_vac]]",

             "row.to_csv('/pfs/out/max_vaccinations.csv', header=None, index=None, sep=' ', mode='a')",

         ],

         image="amancevice/pandas",

     ),

     input=pps_proto.Input(

         pfs=pps_proto.PFSInput(glob="/", repo="data")

     ),

)

print(list(client.list_pipeline()))

You should see the following output:

Figure 11.11 – Create pipeline output

Let's get the result of our pipeline by executing the following code:

client.get_file(("find-vaccinations", "master"), "/max_vaccinations.csv").read()

You should see the following output:

Figure 11.12 – Results of the pipeline

Our pipeline has determined that during the period from December 24 to December 31, the most vaccinations were done in Germany on December 31. The number of vaccinations was 206443 and the manufacturer was Pfizer/BioNTech.

To clean up your cluster, run the following:

client.delete_repo("data", force=True)

client.delete_pipeline("find-vaccinations")

print(list(client.list_repo()))

print(list(client.list_pipeline()))

You should see the following output:

Figure 11.13 – Cluster clean up

In this section, we learned how to perform basic Pachyderm operations in Pachyderm Python Notebooks. Next, we'll create another example pipeline.

Creating and running an example pipeline in Pachyderm Notebooks

In the previous section, we learned how to use Pachyderm Notebooks, create repositories, put data, and even created a simple pipeline. In this section, we will create a pipeline that performs sentiment analysis on a Twitter dataset.

Important note

The code described in this section can be found in the https://github.com/PacktPublishing/Reproducible-Data-Science-with-Pachyderm/blob/main/Chapter11-Using-Pachyderm-Notebooks/sentiment-pipeline.ipynb file.

We will use a modified version of the International Women's Day Tweets dataset from Kaggle available at https://www.kaggle.com/michau96/international-womens-day-tweets. Our modified version includes only two columns—tweet number # and text. The dataset includes 51,480 rows.

Here is an extract of the first few rows of the dataset:

    #                        text

0  3  "She believed she could, so she did." #interna...

1  4  Knocking it out of the park again is\r\n@marya...

2  5  Happy International Women's Day! Today we are ...

3  6  Happy #InternationalWomensDay You're all power...

4  7  Listen to an experimental podcast recorded by ...

Here is a diagram of the workflow:

Figure 11.14 – Sentiment analysis pipeline

In the next section, we will learn about the methodology that is used to build this pipeline.

Pipeline methodology

We will use Natural Language Toolkit (NLTK), which is familiar to us from previous chapters, to clean the data. Then, we will use TextBlob, an open source Python library for text processing, to perform sentiment analysis on the tweets in the dataset.

Sentiment analysis is a technique that helps understand the overall mood of the individuals involved in a specific conversation, discussing specific products and services, or rating a movie. Sentiment analysis is widely used in various types of businesses and industries by marketers and sociologists to get a quick assessment of customer sentiment. In this example, we will be looking at emotions expressed in a selection of tweets about International Women's Day.

TextBlob provides two metrics for sentiment analysis—polarity and subjectivity. Each word in a sentence is assigned a score and then the mean score is assigned to the whole sentence. In this example, we will only determine the polarity of the tweets. Polarity defines the positivity or negativity of a sentence based on a predefined word intensity.

Polarity values range from -1 to 1, with -1 meaning negative sentiment, 0 being neutral, and 1 being positive. If we were to show this on a scale, it would look like this:

Figure 11.15 – Polarity

If we were to put the words into a table and assign them polarity scores, here is what we might get:

Figure 11.16 – Polarity scores

Let's run a quick TextBlob example on a simple sentence to see how it works. Use the code in the sentiment-test.py file to try this example:

from textblob import TextBlob

text = '''Here is the most simple example of a sentence. The rest of the text is autogenerated. This gives the program some time to perform its computations and then tries to find the shortest possible   possible sentence. Finally, let's look at the output that is used for the rest of the process.'''

blob = TextBlob(text)

blob.tags

blob.noun_phrases

for i in blob.sentences:

    print(i.sentiment.polarity)

To run this script, complete the following steps:

You need to install TextBlob in the Pachyderm Notebooks terminal:

pip install textblob && python -m textblob.download_corpora

Here is the output that you should see:

Figure 11.17 – Installing TextBlob

Run sentiment-test.py either in your local terminal or in a Jupyter notebook:

Figure 11.18 – Output of the sentiment-test.py script

As you can see in the output, TextBlob assigns a score for each sentence.

Now that we have reviewed the methodology of our example, let's create our pipelines.

Creating the pipelines

Our first pipeline will use NLTK to clean the Twitter data in our data.csv file. We will create a standard Pachyderm pipeline by using python-pachyderm. The pipeline will consume files from the data repository, run the data-clean.py script on it, and output the cleaned text to the data-clean output repository. The pipeline will use the svekars/pachyderm-ide:1.0 Docker image to run the code.

The first part of data-clean.py imports the components that are familiar to us from Chapter 8, Creating an End-to-End Machine Learning Workflow. These components include NLTK and pandas, which we will use to preprocess our data. We will also import re to specify regular expressions:

import nltk

import pandas as pd

from nltk.corpus import stopwords

from nltk.tokenize import word_tokenize

nltk.download('wordnet')

nltk.download('punkt')

import re

The second part of the script performs the data cleaning using the NLTK word_tokenize method, stopwords with the lambda function, and re.split to remove the URLs. Finally, the script saves the cleaned text to cleaned-data.csv in the output repository:

stopwords = stopwords.words("english")

data = pd.read_csv("/pfs/data/data.csv", delimiter=",")

tokens = data['text'].apply(word_tokenize)

remove_stopwords = tokens.apply(lambda x: [w for w in x if w not in stopwords and w.isalpha()])

remove_urls = remove_stopwords.apply(lambda x: re.split('https:\/\/.*', str(x))[0])

remove_urls.to_csv('/pfs/out/cleaned-data.csv', index=True)

Our second pipeline will perform sentiment analysis on the cleaned data with the TextBlob Python library. The sentiment.py script imports the following components:

from textblob import TextBlob

import pandas as pd

import seaborn as sns

import matplotlib.pyplot as plt

from contextlib import redirect_stdout

We'll use pandas to manipulate the data frames. We'll use matplotlib and seaborn to visualize our results, and we'll use redirect_stdout to save our results to a file.

Next, our script performs sentiment analysis and creates two new columns—polarity_score and sentiment. The resulting table is saved to a new CSV file called polarity.csv:

data = pd.read_csv('/pfs/data-clean/cleaned-data.csv', delimiter=',')

data = data[['text']]

data["polarity_score"] = data["text"].apply(lambda data: TextBlob(data).sentiment.polarity)

data['sentiment'] = data['polarity_score'].apply(lambda x: 'Positive' if x >= 0.1 else ('Negative' if x <= -0.1 else 'Neutral'))

print(data.head(10))

data.to_csv('/pfs/out/polarity.csv', index=True)

Then, the script saves the tweets of sentiment category to its own variable, calculates the total for each category, and saves the totals to the number_of_tweets.txt file:

positive = [ data for index, t in enumerate(data['text']) if data['polarity_score'][index] > 0]

neutral = [ data for index, tweet in enumerate(data['text']) if data['polarity_score'][index] == 0]

negative = [ data for index, t in enumerate(data['text']) if data['polarity_score'][index] < 0]

with open('/pfs/out/number_of_tweets.txt', 'w') as file:

      with redirect_stdout(file):

          print("Number of Positive tweets:", len(positive))

          print("Number of Neutral tweets:", len(neutral))

          print("Number of Negative tweets:", len(negative))

The last part of the script builds a pie chart with the percentages of tweets in each category and saves them to the plot.png file:

colors = ['#9b5de5','#f15bb5','#fee440']

figure = pd.DataFrame({'percentage': [len(positive), len(negative), len(neutral)]},

                       index=['Positive', 'Negative', 'Neutral'])

plot = figure.plot.pie(y='percentage', figsize=(5, 5), autopct='%1.1f%%', colors=colors)

circle = plt.Circle((0,0),0.70,fc='white')

fig = plt.gcf()

fig.gca().add_artist(circle)

plot.axis('equal')

plt.tight_layout()

plot.figure.savefig("/pfs/out/plot.png")

Let's create these pipelines:

Create a new Pachyderm Python notebook.

Create a Pachyderm data repository:

import python_pachyderm

client = python_pachyderm.Client()

client.create_repo("data")

print(list(client.list_repo()))

You should see the following output:

Figure 11.19 – Output of the created repository

Put the data.csv file in this repository:

with client.commit('data', 'master') as i:

     client.put_file_url(i, 'data.csv', 'https://raw.githubusercontent.com/PacktPublishing/Reproducible-Data-Science-with-Pachyderm/main/Chapter11-Using-Pachyderm-Notebooks/data.csv')

print(list(client.list_file(("data", "master"), "")))

This script returns the following output:

Figure 11.20 – Put file output

List the files in the data repository:

list(client.list_file(("data", "master"), ""))

You should see the following response:

Figure 11.21 – List file output

Create the data-clean pipeline:

from python_pachyderm.service import pps_proto

client.create_pipeline(

     pipeline_name="data-clean",

     transform=pps_proto.Transform(

         cmd=["python3", "data-clean.py"],

         image="svekars/pachyderm-ide:1.0",

     ),

     input=pps_proto.Input(

         pfs=pps_proto.PFSInput(glob="/", repo="data")

     ),

)

client.create_pipeline(

     pipeline_name="sentiment",

     transform=pps_proto.Transform(

         cmd=["python3", "sentiment.py"],

         image="svekars/pachyderm-ide:1.0",

     ),

     input=pps_proto.Input(

         pfs=pps_proto.PFSInput(glob="/", repo="data-clean")

     ),

)

print(list(client.list_pipeline()))

You should see the following output:

Figure 11.22 – Create pipeline output

The output is truncated and shows only the data-clean pipeline. You should see similar output for the sentiment pipeline.

Let's view the first few lines of the file in the data-clean repository that has our cleaned data:

import pandas as pd

pd.read_csv(client.get_file(("data-clean", "master"), "cleaned-data.csv"), nrows=10)

This script returns the following output:

Figure 11.23 – Cleaned data

You can see that the text was broken down into tokens.

Let's get the list of files in the sentiment repository:

list(client.list_file(("sentiment","master"), ""))

You should see a lengthy output. The files will be under path:, similar to the following response:

Figure 11.24 – List of files in the sentiment repository

There should be three files, as follows:

number_of_tweets.txt: A text file with the total number of tweets in each category

plot.png: A pie chart with the percentages of tweets in each category

polarity.csv: A new CSV table with the polarity_score and sentiment columns

Now, let's look at the first few rows of the polarity.csv table:

pd.read_csv(client.get_file(("sentiment","master"), "polarity.csv"), nrows=10)

This script returns the following output:

Figure 11.25 – Polarity and sentiment results

You can see the two new columns appended to our original table, giving a polarity score in the range [-1;1] and a sentiment category.

Let's see the total number for each sentiment category:

client.get_file(("sentiment", "master"),"number_of_tweets.txt").read()

You should see the following output:

Figure 11.26 – Total number of tweets for each category

Finally, let's look at the pie chart with sentiment category percentages:

from IPython.display import display

from PIL import Image

display(Image.open(client.get_file(("sentiment", "master"), "/plot.png")))

This script returns the following output:

Figure 11.27 – Percentage of positive, negative, and neutral sentiments

Based on this chart, we can tell that the majority of the tweets contain positive sentiments and the percentage of negative tweets can be considered insignificant.

This concludes our sentiment analysis example.

Summary

In this chapter, we have learned how to create Pachyderm notebooks in Pachyderm Hub, a powerful addition to Pachyderm that enables data scientists to leverage the benefits of an integrated environment with the Pachyderm data lineage functionality and pipelines. Data scientists spend hours performing exploratory data analysis and do so in notebooks. Combining Pachyderm and notebooks brings data scientists and data engineers together on one platform, letting them speak the same language and use the same tools.

In addition to the above, we created a pipeline that performs basic sentiment analysis of Twitter data and ran it completely in a Pachyderm notebook. We have expanded our knowledge of Python Pachyderm and how it can be used in conjunction with other tools and libraries.

Further reading

JupyterLab documentation: https://jupyterlab.readthedocs.io/en/stable/

TextBlob documentation: https://textblob.readthedocs.io/en/dev/

python-pachyderm documentation: https://python-pachyderm.readthedocs.io/en/stable/

Packt.com

Subscribe to our online digital library for full access to over 7,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe?

Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals

Improve your learning with Skill Plans built especially for you

Get a free eBook or video every month

Fully searchable for easy access to vital information

Copy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at customercare@packtpub.com for more details.

At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Other Books You May Enjoy

If you enjoyed this book, you may be interested in these other books by Packt:

Engineering MLOps

Emmanuel Raj

ISBN: 9781800562882

Formulate data governance strategies and pipelines for ML training and deployment

Get to grips with implementing ML pipelines, CI/CD pipelines, and ML monitoring pipelines

Design a robust and scalable microservice and API for test and production environments

Curate your custom CD processes for related use cases and organizations

Monitor ML models, including monitoring data drift, model drift, and application performance

Build and maintain automated ML systems

Getting Started with Streamlit for Data Science

Tyler Richards

ISBN: 9781800565500

Set up your first development environment and create a basic Streamlit app from scratch

Explore methods for uploading, downloading, and manipulating data in Streamlit apps

Create dynamic visualizations in Streamlit using built-in and imported Python libraries

Discover strategies for creating and deploying machine learning models in Streamlit

Use Streamlit sharing for one-click deployment

Beautify Streamlit apps using themes, Streamlit Components, and Streamlit sidebar

Implement best practices for prototyping your data science work with Streamlit

Packt is searching for authors like you

If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Share Your Thoughts

Now you've finished Reproducible Data Science with Pachyderm, we'd love to hear your thoughts! If you purchased the book from Amazon, please click here to go straight to the Amazon review page for this book and share your feedback or leave a review on the site that you purchased it from.

Your review is important to us and the tech community and will help us make sure we're delivering excellent quality content.

Other Books You May Enjoy

Other Books You May Enjoy