Microsoft

Learning Microsoft Azure

Cloud Computing and Development Fundamentals

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

Jonah Carrio Andersson

Learning Microsoft Azure

by Jonah Carrio Andersson

Copyright © 2023 Jonah Carrio Andersson. All rights reserved.

Printed in the United States of America.

Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.

Editors: Jennifer Pollock and Jill Leonard

Production Editor: Beth Kelly

Copyeditor: FILL IN COPYEDITOR

Proofreader: FILL IN PROOFREADER

Indexer: FILL IN INDEXER

Interior Designer: David Futato

Cover Designer: Karen Montgomery

Illustrator: Kate Dullea

November 2023: First Edition

Revision History for the Early Release

2022-02-11: First Release

2022-03-24: Second Release

2022-05-16: Third Release

2022-08-04: Fourth Release

2022-09-12: Fifth Release

2022-11-04: Sixth Release

2023-02-27: Seventh Release

2023-02-27: Seventh Release

2023-05-19: Eighth Release

See http://oreilly.com/catalog/errata.csp?isbn=9781098113322 for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Learning Microsoft Azure, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-098-11325-4

[FILL IN]

Dedication

I would like to dedicate this first book I had ever written to my late and beloved mother, Joan, who died of breast cancer in 2004 when I turned 18. She had her confidence in me since I was a child and her courage inspired me to be strong, resilient, and confident.

I also want to dedicate this book to my dear family, to my husband Gösta, and to my dog Lorentz, who gave all the moral support, and patience and kept me reminded to take breaks from this book project and all the other community engagements that I am passionate to deliver.

This book is also dedicated to my family in the Philippines. I devoted most of my life to taking care of my younger siblings and father, and they are always there to morally support me in every exciting journey I take in life and my career.

Thank you so much Tack så mycket (Swedish) Maraming Salamat (Tagalog)

Preface

Greetings from the Author

Hello, Readers of Learning Microsoft Azure!

As the author, I want to thank you for choosing to read this book.

I appreciate that you picked this book to help you learn about Microsoft Azure. I hope you find this book valuable resource and content for your learning journey in cloud computing and Microsoft Azure. Application development, DevOps, and engineering with cloud technologies have been one of the most exciting experiences in this modern era!

From the old iron age of huge steel mainframe computers to massive virtual modern resources and technologies on the cloud. As much as the industrial revolution forever changed the manufacturing industry and our access to consumer goods, digitization has transformed how we live and work.

Microsoft Azure is a cloud computing platform close to my heart. For the past several years, I have been a software engineer developing applications and DevOps Engineer deploying applications to the cloud. I have gained a lot of learning, hands-on experience, technical insights, and inspiration about Azure.

Azure is a global cloud provider serving millions of organizations, customers, and modern applications. Azure brings powerful benefits not just to the business but to all members of the organization - from the leadership, project managers, clients, and engineering teams.

Building and maintaining applications or enterprise systems is part of my daily work routine. My experience in the software engineering industry and working with the cloud platform Microsoft Azure has helped me broaden my technical knowledge.

My Cloud Migration Journey That Leads Me to You

I was once involved in a cloud migration project to Azure. I had a mission to move an old legacy.NET application hosted on on-premises servers. I had full responsibility for moving it by myself to a cloud computing platform. I was like a superwoman on a cloud migration mission.

From designing the system architecture, re-structuring the databases, fixing the technical debts, fixing data quality issues, programming, and even creating documentation. I put my heart into it, but that cloud migration project was a fiasco, just like some other software engineering projects.

After trying different migration alternatives: lift and shift, refactoring code, and re- architecting the infrastructure, we determined that the system needed to be re-built as though it were new. Unfortunately, that project was stopped just when we were close to success. It was not deployed to production because of the lack of cloud migration strategy, awareness of the significant benefits of using cloud technologyes, the knowledge about the cloud, and many other organizational factors.

It was a project that didn’t make it to the finish line. Still, it was one of the most significant experiences in my cloud engineering career. I learned a lot from that migration to cloud project including the mistakes we made in that project and the lessons learned.

Just like any other project and in life, there are no regrets, just lessons learned.

I hope readers of this book will not commit the same mistakes we did in that project. I do not want other developers or organizations to make the same mistakes we did in that cloud migration project - unprepared for the cloud as a team.

My cloud migration journey as a developer, and partly with the role of being the cloud migration architect, is one of the reasons I am very passionate about sharing knowledge of the cloud computing and Azure, especially to those still starting their journey. I inspire knowledge through public speaking in tech conferences, meetups, etc. I also apply what I teach in the career roles I hold and do every day.

It is a humber honor to spend part of my late nights, after-work hours, and weekends writing this book for you.

I am excited to share vital and valuable information about Microsoft Azure in this book.

I hope that the contents of this book will come in handy in your work, career development, cloud migration journey, and contribution to your organization’s cloud engineering projects.

Why I Wrote This Book

My cloud migration journey as a develop is the primary reason I wrote this book. I want to inspire others to truly plan and get ready for moving their workloads and applications to the cloud with preparation in mind.

When I was studying Computer Science in the 1990s, my first thesis was about how the invention of Internet technology helped local communities and our society. Since then, I have become fascinated with how new technologies allow us in our daily routines and work.

As a Developer, Cloud and DevOps Engineer, I create, build, and develop technical solutions with modern technologies, including the cloud technologies offered by Microsoft Azure. I collaborate in full-stack software development, cloud development, and DevOps.

The more I collaborate with different teams in different areas and the more problems I solve, the more technical skills and knowledge help me be in à jour or up to date along with the evolving modern technologies we have these days.

I am passionate about sharing this knowledge with others - especially those who want to gain fundamental learning of cloud computing and Microsoft Azure. This book aims to help those getting started in cloud computing and Microsoft Azure.

This book will assist IT professionals, project teams, software developers, and cloud engineers choose the appropriate cloud service to use in Microsoft Azure. This knowledge will help solve your organization’s customer use cases and business requirements.

Who Should Read This Book

This book is an essential learning reference book for anybody who wants to learn about the vital cloud concepts and cloud computing services provided by Microsoft Azure, regardless if you are a beginner or on an intermediate level in this field.

This book’s information and technical knowledge will help you plan, design, and develop applications and modern technological solutions and migrate existing workloads and systems to the cloud using Microsoft Azure.

This book is open for anybody with a technical background with career roles like Software Developers, Cloud Engineers, or Cloud Solution Architects. Management or leadership roles in an IT organization, such as IT Project Managers, Technical Sales Managers, and Scrum Masters, would also benefit from learning the concepts of cloud computing in Azure. Also, teams working with traditional and on-premises legacy applications or systems will gain essential insight into designing and developing solutions in the cloud platform through Microsoft Azure service and its technical overview.

This book is also ideal for IT Professionals, Software Developers, and aspiring Cloud Engineers who want to prepare Microsoft Certifications for Azure. Certifications like the AZ-900 Microsoft Azure Fundamentals Certifications, AZ-204 Developing Solutions for Microsoft Azure, AZ-400 Designing and Implementing Microsoft DevOps Solutions, and Exam AZ-305: Designing Microsoft Azure Infrastructure Solutions.

Please note that exam names of the Microsoft Azure certifications may change at any time.

Generally, this book will give you the concepts you need to learn to get started with cloud-native development, administration, data engineering, DevOps, analytics, migration strategies, and other exciting solutions in Azure.

What You Will Learn

By the end of this book, you will be able to gain knowledge of the following:

The essential concepts you need to know about cloud computing and its benefits

The fundamentals of Microsoft Azure as a public cloud provider

The different Microsoft Azure technologies that will help you and your organization develop, transform, modernize, and migrate to modern cloud environments

A comprehensive and helpful overview of the different cloud technologies in Microsoft Azure that will help you choose the right cloud service for your demands, use-cases, software development, and cloud development projects

A jump-start guide on how you can start developing cloud solutions and accelerate your career as an Azure Developer, DevOps, Cloud Engineer, or Platform Engineer

Start developing modern cloud services, applications, and solutions in Microsoft Azure environments using your desired and supported programming languages, frameworks, and tools

Learn about the different cloud security tools, identity and access management (IAM) in Azure

Learn how you can integrate cloud technologies with other services, APIs, and third-party services

Get practical options and learn from best practices on the important things you need to consider when you migrate existing legacy applications to a cloud platform like Microsoft Azure using the different migration tools and services

Get information about the recent tools and cloud adoption frameworks in Azure that will help you in multi-cloud or hybrid cloud environments

Develop with Azure by learning the best practices, technology stacks, and tools you can use to build cloud-native applications and solutions such as Infrastructure as Code (IaC), Serverless, Containers, and other cloud technologies in Azure

Learn By Doing Azure

Learning Microsoft Azure is your guide as you work with Microsoft Azure. I always believe in Learn By Doing way of learning.

Check out a blog post I wrote about this topic: Continuous Learning — An Integral Part of A Programmer’s Development

So, invest time in learning the fundamental concepts and doing some hands-on.

Navigating This Book

PART I Fundamentals of Cloud Computing and Microsoft Azure

In this book’s introductory part, you will learn about Cloud Computing and Microsoft Azure Fundamentals.

Chapter 1 provides an introduction to the essential concepts of cloud computing, how it works, the different types of deployment models, understanding the types of cloud, what is CapEx and OpEx in cloud computing, and the benefits of utilizing cloud computing in businesses, IT organizations, society, and software engineering.

Chapter 2 focuses more profoundly on the theoretical and technical concepts of Microsoft Azure as a public cloud platform. Learn about the Microsoft Azure core components and the different cloud services categorized by their purpose.

By the end of reading Part I, you have already gained vital knowledge and foundational concepts of Cloud Computing and Microsoft Azure.

PART II Cloud Computing Services, Networking, Storage and Databases in Microsoft Azure

This second part of the book and its chapters focus on giving you an in-depth of the different technologies in Microsoft grouped into its categories..

Chapter 3 explores some of the Microsoft Azure compute services such as Azure Virtual Machines, container services like Azure Container Instances, Azure Container Registry, Azure Container Apps, Azure App services for web and mobile applications, serverless cloud solutions with Azure Functions, Azure Static Web App, and more.

Chapter 4 covers cloud networking and services in Microsoft Azure, including Azure VNet, DNS, Azure Firewall, Azure Front Door, ExpressRoute, Virtual Network, VPN Gateway, Application Gateway, Load Balancer, Internet Analyzer, and more. This chapter briefly mentions Azure Orbital, a fully-managed Ground Station as a Service (GSaaS) solution in Azure.

Chapter 5 provides a technical overview of the different cloud storage and databases (both SQL and NoSQL) in Microsoft Azure. You will learn about cloud storage concepts and services, create databases and find applicable best practices for Azure SQL Databases, Azure Cosmos DB, Database for MySql, Azure SQL Servers, Redis Cache in Azure, Azure Storage, Data Share and Manage Disks.

PART III Artificial Intelligence (AI), Machine Learning (ML), Big Data, Analytics, Internet of Things (IoT) and Security in Microsoft Azure

Chapter 6 focuses on the valuable concepts you need to know about Artificial Intelligence(AI), Machine Learning, Azure OpenAI Service, Cognitive Services in Microsoft Azure. This chapter also covers the importance of responsible and ethical AI.

Chapter 7 explores the Big Data, Reporting, and Analytics Services in Microsoft Azure. This chapter will include what you need to know about data analytics, big data, analytics, and reporting services using Power BI, Azure Stream Analytics, Data Lake Analytics, Azure HD Insights, and Azure Analysis Services. This chapter will help you gain the critical concepts in working with extensive complex data using Azure services and big data tools.

Chapter 8 covers the Azure solutions for IoT (Internet of Things), Maps Services, and Cognitive services in Azure. You will learn about Azure IoT Hub, IoT Edge, Azure Maps, Azure Spheres, and Remote Rendering services. You will learn how to use some of the Azure IoT kits available for developers such as Azure Developer IoT Dev Kit to start IoT development with Microsoft Azure.

Chapter 9 dives into identity and access management(IAM), compliance, cloud security, and DevSecOps in Microsoft Azure which is important in protecting and securing your applications and cloud workloads.You will learn about Azure Active Directory and Azure AD B2C, B2E. Azure security services like Azure Key Vault, Azure Sentinel, Azure Firewall, Microsoft Defender for Cloud and its sub-features, Azure security for networking, data, compute, and security monitoring will be explored.

By the end of reading Part III, you have already learned some technical knowledge and hands-on experience with the different technologies in Microsoft Azure. These will help you choose what Azure technology to use for what you need in your business requirements and develop solutions with them.

PART IV Cloud Integration Services, Cloud Infrastructure, Cloud Development and DevOps in Microsoft Azure

This book’s fourth section will teach you how to integrate Azure technologies with other services, Infrastructure as a code solution, and set up your application using Azure’s deployment technologies.

Chapter 8 focuses on Microsoft Azure integration services. In this chapter, you will learn about the different cloud technologies you can integrate with your cloud-native applications, including the existing ones (on the cloud or on-premises). This chapter covers Azure API Management, Azure Logic Apps, Azure Web PubSub, Azure Service Bus, Azure Event Grid, Azure Logic Apps, etc.

Chapter 11 guides on how to develop systems or applications with modern DevOps tools today. You will learn what DevOps means, the comprehensive suite like Azure DevOps for team collaboration for developers and IT operations. You will also know about automating development processes using CI/CD and source code version control, what Azure Pipelines are, Github, and Azure DevOps, and how Azure Monitor and Application Insights are used for troubleshooting Azure resources using Application Insights. You will also learn about other cloud technology services like Azure DevTestLabs, Azure Bicep, ARM Templates, and more. Know what Infrastructure as a Code (IaC), Policy as a Code, Configuration as a Code,and how they help with the automation of deployments.

After reading the chapters in Part IV, you already have gained knowledge and development skills to help you work effectively as a cloud developer using the cloud integration and automation options you have. You and your agile team in your organization will also be able to collaborate and work effectively using the great features of Azure DevOps, DevTest Labs, GitHub, etc.

PART V Cloud Management, Governance, Migration Tools and Architecture in Microsoft Azure

The fifth section of this book will give you essential knowledge and tools into adopting and migrating successfully to the cloud.

Chapter 12 walks you through the essential concepts you should know regarding cloud management and cloud governance in Microsoft Azure. For example, it works with automation in the cloud, Azure Advisor, backup using Azure Backup, Azure Blueprints, Azure Policy, Azure Monitor, and other known solutions for hybrid and multi-cloud.

Chapter 13 discusses cloud migration, cloud transformation, and architectural concepts in Microsoft Azure. You will learn the vital facts you need to know when adopting, transforming, or migrating to Azure. Find out the best practices and helpful list of tools you can use when moving on-premises and legacy old applications. This chapter also highlights the importance of having Microsoft Azure’s Well-Architected Framework.

After reading the chapters in Part V, you have already gained an understanding and essential knowledge of Microsoft Azure cloud governance and management. These are important in designing and developing cloud solutions. You will also capture a broad understanding of the Microsoft Cloud Adoption Framework (CAF), Well-Architected Framework for Azure (WAF), Azure Migrate, and Microsoft Assessment tools that can assist your cloud migration projects. You will also have learned from lessons I learned from a cloud migration project experience in migrating old .NET Legacy On-Premise applications to Azure.

PART VI Cloud Development and Continuous Learning with Microsoft Azure

The fifth and final section of useful list of resources.

Chapter 14 is more focused on the cloud development with Azure. This chapter aims to guide you, the reader, to the different language and cloud development tools that you can use to get started with developing modern and resilient applications and solutions on Azure. This chapter covers IDE tools such as Visual Studio, VSCode, Azure Cloud Shell, Github Co-pilot, Codespaces, Microsoft Dev Box, and many more. You learn the different technologies that are good to learn and prepare for in your projects and career development.

Check Self-Knowledge

In the end of each chapter mentioned, I will be providing a short list of Check Your Knowledge section with a maximum of 5 learning check questions related to the topics discussed. These questions will be a useful review challenge for you to check what you have learned.

Learn By Doing (Try it out!)

As a developer, I love the concept of learn by doing. I believe in theoretically learning enhanced by hands-on practices. Therefore, each chapter in this book also ends with a section called Learn By Doing (Try it out) where I provided a list of quickstart guides and how-to links to get started with hands-on learning related to the topic discussed in the chapter.

Learning Resources and Further Readings

Learning Microsoft Azure aims to provide the most important foundational knowledge you need to know. With consideration that we have different levels of experience and knowledge, you probably want to explore in more depth the discussed topic further. Therefore, on the end of each chapter, I will be providing a short list of recommended learning and further reading resource related to the topics discussed. Most the references are linked to the most recent documentation of Microsoft for specific technology or resource mentioned in this book.

What This Book Is Not

This book is not an advanced level book for hands-on for each cloud technology service in Microsoft Azure. Hands-on examples are provided but not intented to those who already worked with Azure at highly advanced level.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

Tip

This element signifies a tip or suggestion.

Note

This element signifies a general note about the topic discussed.

Warning

This element indicates a warning or caution.

Using Code Examples

Supplemental material (book information, code examples, exercises, etc.) will be available for download later at https://github.com/learningazurebook/.

If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com.

This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Book Title by Some Author (O’Reilly). Copyright 2012 Some Copyright Holder, 978-0-596-xxxx-x.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com.

O’Reilly Online Learning

Note

For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed.

Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

O’Reilly Media, Inc.

1005 Gravenstein Highway North

Sebastopol, CA 95472

800-998-9938 (in the United States or Canada)

707-829-0515 (international or local)

707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://learning.oreilly.com/library/view/learning-microsoft-azure/9781098113315.

Email bookquestions@oreilly.com to comment or ask technical questions about this book.

For news and information about our books and courses, visit http://oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments

I want to acknowledge and thank those who helped, supported, and contributued in making this book a success from start to finish. This book would not have been made possible without my family, friends, and the incredible people in the tech and cloud communities who helped and contributed.

Special thanks to my friends in the community who also helped in the quality of my book by being the technical reviewers of this book. I appreciate the time invested by technical reviewers Peter De Tender, Håkan Silfvernagel, Micheal John Peña, Nikos Delis, Stefano Demiliani, Peter Lee, Sagar Rastogi, Vladimir Serykh, Szabó Attila, Ian Santillan, George Grammatikos, Dipal Choksi, Alexander Wachtel, Dave McCollough, George Mount, and the rest of the reviewers.

To Andrew Urwin and Freek Berson who partly contributed their input about working with DevOps and infrastructure as code (IaC) with Bicep Language in Azure in the Chapter 11.

Thanks to experts John Kilmister, Alexey Polkovnikov, Ryan O’Connell, Håkan Silfvernagel, Adrienne Braganza Tacke, Goran Vuksic, Andrew Urwin, Ezhilarasi Chezhiyan, George Mount, Tiago Costa, Marilag Svennevig, and many others who provided their expert quotes at the beginning of each chapter.

Thanks, Jill Leonard, Jennifer Pollock, Megan Laddusaw, Elizabeth Kelly, Suzanne Huston, Rita Fernando, and the rest of the O’Reilly Media team.

Special mention to who helped me through the journey of this book.

NOTE: Acknowledgement is still on draft and to be updated until the book is finished.

Part I. Fundamentals of Cloud Computing and Microsoft Azure

In this introductory part of the book, you will learn about the Cloud Computing and Microsoft Azure Fundamentals.

Chapter 1 provides an introduction to the important concepts of cloud computing, how it works, and the different types of its deployment models. This chapter gives you a general perspective and understand the types of cloud computing. Know what is the difference between CapEx and OpEx in cloud computing, and the benefits of utilizing it in businesses, IT organizations, society and software engineering.

Chapter 2 focuses deeper into the theoretical and technical concepts of Microsoft Azure as a public cloud platform of Microsoft. In this chapter, you will learn about the Microsoft Azure core components and the different cloud services categorized by its purpose.

By the end of reading Part I, you already have gained vital knowledge and foundational concepts of Cloud Computing and Microsoft Azure.

Chapter 1. Cloud Computing Fundamentals

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 1st chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at jleonard@oreilly.com.

Over the last decade, cloud computing has changed the way we build and deploy software. It is now easier than ever to deploy highly scalable, resilient, and secure solutions to a global audience, in many cases at a fraction of the previous cost.

John Kilmister, Software Architect and Microsoft Azure MVP

What is Cloud Computing

Before delving into learning Microsoft Azure, it’s essential first to understand cloud computing. Cloud computing is a significant technological innovation that delivers various services through the Internet, including web servers, databases, data storage, virtual machines, applications, network infrastructure, security tools, software, and other IT infrastructure. “cloud computing” refers to the virtual storage and access of data and information over the Internet, with the actual computing processing transpiring in the cloud.

Understanding the foundational principles of cloud computing is essential for designing and developing solutions in the cloud. As a public cloud platform, Azure provides secure, scalable, reliable, cost-effective, and easy-to-manage ways to build on the cloud. By leveraging Azure, organizations and users developing with it can take advantage of various cloud services and tools to innovate and solve complex business problems.

Cloud computing is a game-changing technology that enables businesses to access a wide range of IT resources on demand through the Internet. With Azure, organizations can build and deploy cloud-based solutions that are secure, scalable, and cost-effective.

John McCharty’s Interesting Prediction of Cloud Computing

Looking back into history, John McCarthy was an American computer scientist, and cognitive scientist who was known as the father of Artificial Intelligence, gave some interesting insights about cloud computing in his speech at MIT’s centennial celebration in 1961. He suggested that computing can be sold like a public utility, just like water or electricity.

Cloud computing makes life easier for us. A good and practical example is the possibility of saving our photos, videos, and files into cloud storage virtually with unlimited capacity instead of keeping them on a local storage device with limited storage. Another benefit is virtualizing web servers and databases instead of having the physical infrastructure or servers in costly data centers.

Software engineering and modern I.T. innovation are a few significant advantages of the cloud. Adopting and implementing cloud computing provides a fully-managed cloud computing infrastructure and services with the benefits of scalability, autoscaling, availability, and performance at a flexible global scale.

The different cloud computing solutions give us tools and capabilities to handle peak loads based on demands at any time on a global scale. This is complicated, expensive, and time-consuming in an on-premises environment. Azure seamlessly handles scenarios like this through horizontal or vertical scaling or a combination of both.

As you read through the chapters in this book, you will be able to relate and learn how cloud computing technologies help you and your organization in different areas.

Benefits of the Cloud in Software Engineering and IT

Cloud computing and software engineering are both evolving rapidly. The evolution of computing in the cloud, along with innovations like Machine Learning (ML), Internet of Things (IoT), Edge Computing, Quantum Computing, and Big Data, has caused an increased demand for skillsets and people who can skill in these technologies and platforms in the cloud.

Computing in the cloud enables software developers, engineers, and even I.T. professionals to create, build, test, and deploy technical cloud solutions productively, effectively, and securely. Software engineering teams still need to be expanded to work and develop on-premises and encounter technological gaps. They may experience the risks of missing the advantages of developing cloud computing systems.

Engineering teams, business teams, and organizations may risk missing the great features, benefits, and capabilities that cloud computing provides. Cloud engineering offers better speed of development, testing, maintainability, automation, scalability, and so much more.

With the advancement of modern automation processes like Infrastructure as a Code (IaC) ecoming available for cloud infrastructures, benefits such as infrastructure automation are making software development easier for developers and DevOps teams. Through the IaC approach, there will be better consistency and routines for configuring systems with the capability of replicating systems to several environments.¹.

In addition to this, solutions and platforms using Low-Code/No-Code like Power Apps, Azure Logic Apps, AI Builder, etc., on the cloud also help IT professionals with less programming skills and experience to build modern and smart applications quickly on demand.

Infrastructure as code technologies and low-code/no-code solutions will be discussed in technical details later in this book.

In complement to the benefits the cloud has for engineering teams, I.T. project managers can easily manage their projects and collaborate with their teams by working agile with available modern and remote collaboration tools in the cloud. For example, Microsoft Azure DevOps is a DevOps all-in-one collaboration suite.

Azure DevOps helps cover the entire application lifecycle. From agile project planning, source code versioning, continuous integration, and continuous delivery (CI/ CD), testing plans, artifacts, and integrations. We will dive more into Azure DevOps, cloud development integration tools, infrastructure management, etc., in Chapter 11 of this book.

Overall, the innovation brought by cloud computing has helped us advance in our society and improve our daily lives.

Digitalization and modernization come with great benefits; however, it also comes with their challenges. These include the challenges of preparing, transforming, and adapting to fast-changing and evolving technologies. These barriers can be handled by learning the foundations of cloud computing, which you will learn by reading through this first chapter.

Figure 1-1. Overview of Cloud Computing

Instead of having our resources like databases, applications, servers, or infrastructure physical data centers or on-premises, we have these resources on the cloud or Internet, as shown in Figure 1-1.

What are Data Centers?

In cloud computing, a data center is a physical facility that houses many servers and other computing equipment. It is designed to provide a centralized location for storing, processing, and managing large amounts of data and applications that can be accessed remotely through the Internet.

Data centers are the backbone of cloud computing, providing the infrastructure necessary to deliver cloud-based services, including storage, computing power, and network connectivity.

The physical design of a data center is carefully planned to ensure high availability, security, and energy efficiency. They are equipped with redundant power and cooling systems, backup generators, and other measures to ensure continuous operation, even during a power outage or further disruption. They also employ strict security measures, such as biometric authentication, firewalls, and intrusion detection systems, to protect the data and applications.

Businesses and organizations consider cloud computing technology as a good and strategic option because of its speed, reliability, financial savings, productivity, efficiency, security, performance, and more. By the end of this chapter, you will learn the specific benefits cloud computing has for different categories.

Cloud computing has been a popular option for many these past years because of the growing demand and evolution of technological innovations over the past decades. Organizations want to improve and modernize their systems to innovate with the new trends in technology.

At the same time, they also need to strategically consider the value of cost savings and think about the increased productivity of their teams, global scale opportunities, efficiency, performance, and security. Computing in the cloud is expanding and continues to grow our mission to sustainability is also one of the great drivers of cloud innovation.²

How Cloud Works? Cloud Computing vs. Virtualization

Cloud Computing and virtualization both create useful virtual environments. Hosting compute and data resources on the cloud is better than a virtual machine for a web server or databases hosted on an on-premises environment. Hosting in the cloud provides efficiency, flexibility, and security.

The cloud is an environment, while virtualization is a technology. Virtualization is a technology that enables us to virtualize hardware to create and simulate several machines or dedicated resources. On the other hand, the cloud is an I.T. environment that pools and shares scalable resources across a network. Cloud environments are created to activate the extraordinary capabilities of cloud computing, like running workloads within it.

Environment for Cloud Computing

Cloud computing is a modern approach to providing on-demand resources for computing, storage, network, platform, web applications, and infrastructure over the Internet or cloud. These are pools of virtual services and resources hosted on the cloud, which is accessible by its users anywhere in the world. Cloud administrators can manage these cloud resources through self-service administrative portals—for example, Microsoft Azure Portal for Microsoft cloud resources.

Cloud Hypervisor - The Key to Virtualization in the Cloud

The powerful hypervisor technology is emerging to be a vital tool in virtualizing resources and is driving modern innovation in cloud environments. Hypervisors make resources and applications in the cloud available to their users remotely. Accessing and managing cloud resources over the Internet gives organizations and I.T. better control in managing their systems, applications, data, and infrastructure in the cloud environment. Hypervisor technology is illustrated in Figure 1-2 is a simple example.

The emerging transformation to digitalization and rising demand for better service expectations of customers are resulting in the building more modern and reliable applications. For such reasons, organizations are still considering migrating or are already migrating their enterprise applications from on-premises virtual machines to cloud environments.

An excellent example of the use of hypervisors in cloud computing is the Microsoft Azure cloud platform. Microsoft uses a native hypervisor on Azure called Azure Hypervisor. It enables deployments of virtualized machines, web servers, database servers, enterprise applications, web services, etc., on the cloud. The hypervisor is based initially on Windows Hyper-V. It provides various virtualization deployment, management, monitoring, and security features.

Figure 1-2. Hypervisor Technology

Nevertheless, even with all of the benefits we’ve just described, migrating existing enterprise applications to the cloud is a challenging quick-fix journey. It requires careful planning, implementation of good strategy, and more resources to re-architect, re- write systems or applications for cloud upgrade. Through the technology of virtualization with the hypervisor, it is possible to migrate existing on-premise workloads to any cloud computing platform faster while investing less time, money, and resources.

The Technology of Virtualization

Virtualization technology simulates resources and environments from a single physical hardware system. Behind this technology is the Software emulated called hypervisor has the capability to distribute a system into secure and distinct environments known as Virtual Machines (V.M.s). The virtual machines (V.M.) we use on our traditional web servers these days rely on the ability of a hypervisor. V.M.s are emulations of computers running on top of a hypervisor.

Today, containerization is one of the better alternatives to virtual machines. Unlike a virtual machine, a container is a lightweight, portable, and isolated unit of Software that enables us to run multiple containers on a single host machine. Containerization is a new technology that allows us to run applications and services on the cloud.

In the later chapters of this book, you will learn more about Azure’s different virtualization, infrastructure, container solutions. In Chapter 13, you will learn more about Microsoft Azure cloud migration concepts and solutions.

Evolution of Cloud Computing

Earlier computing technologies were mainframe computers which provided extensive computational capabilities. Mainframes were powerful, highly reliable specialized for large data movements and massive I/O operations. Large organizations mostly use them for bulk data processing. Mainframes worked on batch processing.

There were different stages of earlier computing before we started using the modern and dynamic cloud platforms like Microsoft Azure. However, the early mainframe computer systems are similar to the modern cloud computing platforms.

For example, both of them use a client-server model and thin clients. The significant difference why many organizations and institutions are migrating to the cloud is because of the cost savings, increased productivity of I.T. teams, speed, availability, flexible scaling, efficiency, performance, and security.

The earlier cloud computing technologies have evolved through time to create today’s more dynamic technology solutions and offerings like the public cloud, such as Azure, provides.

As shown in Figure 1-3, the different earlier computing have evolved since the big mainframe computers in the 1960s to our modern and dynamic cloud computing today.

Figure 1-3. Evolution of Cloud Computing

Mainframe Computing

Mainframe computing is considered a high-performance computer mainframe. It is also known as big iron that uses a single unit hardware like a huge mainframe box with several processors, centralized storage, and large memory. IBM is the pioneer of mainframe computers that are mainly client/server-based systems. It is known to have a high performance and processing power to process massive data like transactions and calculations in real time.

A mainframe has the characteristics of being time-shared, highly secured, and supports batch processing. The drawbacks of the mainframe are that they are expensive hardware to maintain and it does not support X86 architecture. A significant challenge of mainframe computers is the limitation of skilled engineers who can maintain these big iron computers.

Cluster Computing

As shown in Figure 1-4, Cluster computing consists of tightly coupled computers (also known as Nodes) that work together to reach a single goal and purpose- to execute tasks. A cluster’s components are connected through a closed group of local area networks(LANs). When multiple computers are clustered, they share the computation tasks like a distributed system.

Figure 1-4. Example of Cluster Computing Architecture

Several types of cluster computing are commonly used for implementations of business requirements, optimization of performance, high availability clusters (H.A. Clusters), high-performance clusters (H.P. clusters), and load-balancing clusters.

Cost-effectiveness, scalability, high availability, and speed processing are some of the benefits of using cluster computing. It can be implemented in real-life use cases like search engines, earthquake simulation and weather systems.

The earthquake simulation is exciting and valuable because the earthquake dynamics is significant and challenging in geophysics and computer modeling because of its highly non-linear nature.

To learn more about these interesting earthquake simulation studies with the help of cluster computing, please read NaradaBrokering: A Distributed Middleware Framework and Architecture for Enabling Durable Peer-to-Peer Grids and Study Uses Supercomputers to Advance Dynamic Earthquake Rupture Models.

Cluster computing is composed of multiple computer systems considered nodes. All these nodes are used altogether to execute tasks. This type of computing has expanded considerably in our modern days.

Azure has high-performance computing (HPC) which includes a great set of integration of resources from storage, computing, and networking with workload orchestration ideal for HPC systems.

Grid Computing

Grid computing is a subset of parallel and distributed computing in which clusters of computers and a loosely coupled computer performs a big task. The computer resources can be geographically spread out in different locations or in several computing clusters that form acts as the grid. The advantage of this is that data is processed quickly with speed because the data are stored on all computers in the data grid, as shown in Figure 1-5.

Figure 1-5. Grid Computing Example - A Grid Resource Management System

The computer systems on the grid in the same network work together and serve as a virtual supercomputer. All systems in the grid use compute resources like storage capacity and processing power.

Cloud Computing

Cloud computing has emerged and is considered fifth-generation computing. The evolution of mainframe computing, grid computing, and cluster computing created an enormous path of accelerated innovation that drove and enabled the modern computing we have today.

The technology of cloud computing, as shown in Figure 1-6, is widely used nowadays, and we are continually exploring more of its capabilities in our modern digitalization.

Based on the Cloud Computing Study 2022 by Foundry, over 40% of the companies are planning to migrate their data integration, disaster recovery, Business Intelligence (B.I.), Data warehousing, Data Analytics, and back up to the cloud.

Figure 1-6. Cloud Computing Overview

Our Journey to use Modern Cloud

Modernization and digitalization are the most impressive and impactful innovations in our history. When I was studying Computer Science in the 1990s, I used a lot of small-capacity storage floppy disks to save my documents and photos.

In contrast to what we have today, using the cloud, we have many alternatives and an enormous capacity to collect and store data. The data we store on the cloud is portable and accessible anywhere. Portability of accessing data and getting the information we need when we need it, wherever we are, gives a huge list benefits and is also practical.

We live in the days of modern cloud computing that delivers reliability, scalability, agility, cost savings, portability to our applications and resources globally. Azure compute services which we will dive deeper in Chapter 3, enables us to build, manage, and scale cloud computing applications and services.

Different Types of Cloud Computing Deployment Models

The different cloud computing deployment models give us a descriptive overview of cloud computing platforms in their various categories and parts. It also helps identify essential facts like who has access to it, how it is hosted, and what is implemented.

Figure 1-7 shows that different cloud deployment models are commonly used - Public Cloud, Private Cloud, and Hybrid Cloud. Other deployment models, such as Community Cloud and Multi-Clouds, have been trending and in demand these days. These deployment models work the same way by virtualizing servers’ computing power into segmented‐ ted applications with speed, reliability, scalability, and storage capacities.

Figure 1-7. Different Types of Cloud Computing Models

Public Cloud

Public Cloud is a deployment model where the cloud infrastructure is available to the public or any organization using or selling cloud services. A public cloud platform is a service provided by cloud providers like Azure. The public cloud vendors offer cloud storage and computing resources (Operating Systems, CPU, Memory, Storage, Web Servers, Applications, or Databases) that are securely shared among its customers with other organizations or tenants of the cloud. Using the public cloud is usually offered to its users for a subscription fee or Pay-As-You-Go basis.

For example, in Azure, you can already start using that platform as an individual by signing up for a free account with Pay-As-You-Go type of subscription with included set of free Azure services.

Figure 1-8. An example of several organizations using Public Cloud

Many organizations globally have adapted and evolved to use the public cloud as their central platform for IT infrastructure and services, as illustrated in Figure 1-8. In particular, organizations with the business requirement of having the workloads of their applications in the public cloud can use this type of cloud.

The Advantages of using Public Cloud

We are in the modern era of technology with rapid acceleration around improvement and innovation. Along with this continuous shift towards digitalization, organizations must focus on delivering quality products and solutions to their customers. Organizations also want to be competitive in the global market by going beyond geographical barriers. They want to modernize their solutions and services by investing in large-scale cloud solutions.

Whether your business is focused on cost reduction, global scale, better administrative management, or wants modern solutions with enhanced security, there are many great benefits to moving to the public cloud. A list of some of the services available by migrating to the cloud is as follows.

Cost Effectiveness and Cost Management

On-Demand Services and Portability

Scalability and Reliability

Sophisticated and modern solutions

Flexibility in administration through self-service cloud management portals

Monitoring, Analytics and Reports Visualization

Resource Pooling

Security and Privacy

Disaster Recovery and Geo-location

Private Cloud

A private cloud is a type of cloud infrastructure that is operated and owned by one organization - on-premise or off-premise. An organization utilizing private clouds uses cloud computing technology with considerations to privacy and the importance of security. This means that access to the resources in the IT infrastructure within the organization is centralized. Trust boundaries define the organization’s administration of private clouds.

Figure 1-9. Private Cloud Example

In private clouds as shown in Figure 1-9, the infrastructure and its resources are typically managed on the organization’s private cloud on a virtual private network. Organizations that use the private cloud in their IT infrastructure are responsible for managing and maintaining their infrastructure.

Some institutions with special requirements and IT policies requiring enhanced security and control over the cloud infrastructure use private clouds. Financial institutions, government agencies, and organizations that require advanced security and strict privacy usually prefer this cloud.

Let us take an overview of that what the known advantages of using Private Cloud

Enhanced privacy and security since resources are not shared with others

Increased control over the infrastructure and owned resources

Compliance to business-critical security and regulatory compliance requirements

Flexibility to customize the environments based on the on-demand requirements of the organization or business

Community Cloud

Community Cloud is a hybrid form of the private cloud. They are multi-tenant platforms that enable different organizations to work on a shared platform. This type of cloud is rarely mentioned publicly, but it is used widely and exists. The community cloud is unique and special with its collaborative purpose.

The following are some examples of institutions and organizations that uses community cloud:

Government

Most cloud providers offer community cloud for governments; they can be known as cloud for the government. For example, cloud provider Amazon AWS Cloud Computing for Federal Government while Microsoft has Azure Government for US government agencies and their partners. Clouds for the government sector are community clouds meant explicitly for the government since they have legal, security, and private regulations, processes, and services that require constant communication and data transactions between different departments. They all operate on the same infrastructure, with benefits and shared resources.

Healthcare

Healthcare sector in the U.S. is regulated by Health Insurance Portability and Accountability Act of 1996 (HIPaA) compliance. This compliance holds and controls the security and transfer of sensitive patient record information. Sensitive patient information includes medical records exchanged between hospitals and laboratories. Healthcare sectors adopting the cloud are using community cloud providers that adhere to HIPAA regulations. Aside from compliance, the healthcare sector is also adopting cloud computing technology to improve healthcare services and costs using artificial intelligence and machine learning.

Education

When COVID-19 struck and affected the entire world and many countries, it affected the education sector and institutions in many countries globally. Because of the pandemic regulations, schools were challenged to deliver education in person. Internet, cloud computing, and remote access to education and learning materials have helped schools and universities develop online education for students in some countries. Community clouds for education have helped many students continue their teaching and learning. Azure for Education is a cloud provider that offers cloud services such as Azure Lab Services, Teams, Office 365 Educational Plans, Azure Virtual Desktop for education.

Remote and Hybrid Work

Based on a recent statistics and study, about 16% of companies globally are 100% remote while 77% of the remote works claim that they are more productive when they are working from home. Cloud computing technologies are being used a lot lately for remote and hybrid work ³, and they will continue to change how we work and collaborate. Some companies enable remote employees to securely connect to their systems from any device over any network. Organizations also use community clouds for innovations governed by regulations before hosting in the public cloud. This means community clouds are being used as a first initial setup hosting resources and infrastructure to a private cloud.

The infrastructure of the community cloud supports and aims for a specific community with shared missions, compliance, security, jurisdictions, etc. Community cloud, as shown in Figure 1-10 can be managed by a community or organization that can be controlled or hosted internally or externally.

Figure 1-10. Community Cloud Example Illustration

Hybrid Cloud

As its name describes, a hybrid cloud is a type of cloud infrastructure composed of multiple clouds, a combination of private, public, or community clouds, as shown in Figure 1-11. In hybrid clouds, unique entities are kept but bound together by standardized technology, which allows the portability of applications and data. For example, load-balancing between clouds through cloud bursting.

Figure 1-11. Hybrid Cloud Example

Cloud bursting is common in hybrid cloud scenarios, as shown in Figure 1-12. It is a technique in application deployment in which an application runs in an on-premises data center or private cloud. Then it can burst into a public cloud if the workloads or computing capacity demands increase. This technique of cloud computing grants accesses to more computing resources when needed.

A few known benefits of cloud bursting are its agility and ability to adjust to rapidly changing workloads. It also provides a cost-effective way to scale up and down computing resources. One practical example is the flexibility to handle the compute workload issues by re-routing the traffic from a private cloud and expanding or “bursting” it to the public cloud.

Figure 1-12. An illustration of Cloud Bursting in the Cloud

What is Multi-Cloud?

Multi-cloud refers to using multiple cloud computing services from different providers to meet a business’s needs. In other words, rather than relying on a single cloud provider, an organization uses services from multiple providers, such as Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), or IBM Cloud.

Let us learn some of the benefits of implementing a multi-cloud approach:

Reduced Risk of Downtime

By using multiple cloud providers, organizations can reduce the risk of downtime or service interruptions. If one provider experiences an outage, the organization can quickly switch to another provider to keep its services running.

Cost Optimization

Organizations can use different cloud providers for other purposes based on pricing, performance, and features. This can help optimize costs and prevent vendor lock-in.

Improved Security

Multicloud can enhance security by reducing the possible risk of a single point of failure. Furthermore, organizations can use different providers for different security needs, such as those with specialized security features for sensitive data.

Flexibility

It helps organization be more agile and flexible, enabling them to adapt to changes in the market and their business needs. For example, they can choose a provider based on their specific requirements for a particular project.

Better Performance

By using multicloud strategy for IT infrastructure can help organizations achieve better performance by leveraging different providers’ strengths and using the right provider for the right workload.

In summary, multi-cloud is a strategy that enables organizations to use multiple cloud providers’ strengths to meet their business needs while improving flexibility, cost optimization, security, and performance.

Hybrid Cloud vs. Multi-clouds

Hybrid and multi-cloud are two different cloud computing architectures, each with its own set of benefits and considerations.

A hybrid cloud is a computing model combining public and private cloud resources to offer a comprehensive solution. This approach enables organizations to maintain control over their sensitive data by keeping it in their private cloud while also taking advantage of the scalability and cost-efficiency of public cloud resources.

Organizations can achieve greater flexibility in managing their workloads and data by using a hybrid cloud. They can use the private cloud for workloads requiring higher security and control levels. In contrast, the public cloud can be used for workloads that demand greater scalability and cost-effectiveness. A hybrid cloud approach offers a more adaptable and effective solution for organizations that require both security and flexibility in their cloud computing environment.

Benefits of hybrid cloud include the following:

Improved Security

Organizations can keep their most sensitive data in the private cloud and still leverage the scalability and cost-efficiency of public cloud resources.

Greater Flexibility

Hybrid cloud allows for more flexibility in managing workloads, as it allows organizations to choose the cloud environment that best suits their needs for each individual workload.

Cost Efficiency

Hybrid cloud can reduce costs by allowing organizations to leverage the cost advantages of public cloud resources while keeping mission-critical data in the private cloud.

On the other hand, in a multi-cloud architecture means the use of multiple public cloud providers for business strategy purposes. In this environment, an organization can use different cloud providers to meet different needs. For example, one provider might offer better storage solutions, while another might have better data analysis tools.

Let us take a look at the common use and benefits of multi-cloud:

Avoiding Vendor Lock-in

Multi-cloud can help organizations avoid being tied to a single cloud provider and are subject to their pricing and policies.

Increased Resilience

Multiple cloud providers can provide additional redundancy and backup options to ensure business continuity and disaster recovery.

Agility

Multi-cloud allows organizations to choose the best cloud provider for each workload, ensuring that each workload has the best resources and capabilities available.

While hybrid and multi-cloud are different, both offer significant benefits in terms of flexibility, security, resilience, and cost-efficiency. The choice between the two depends on the specific needs and objectives of the organization.

Accordingly, it is important to understand the differences between these types deployments so that you plan well how you would design your cloud architecture and infrastructure. Being aware of the benefits and limitations of hybrid and multi-cloud would be ideal for an organization’s cloud strategy.

Public Cloud Computing Providers

This book is about learning Microsoft Azure; however, since we are learning about cloud computing and multi-cloud in this chapter, it is crucial to learn about the other public cloud providers in the market too.

Migrating on-premises applications or systems to the cloud is a challenging process. It requires serious planning, strategy, and preparation.

It is difficult to say that one cloud provider is better than the other. However, choosing the appropriate cloud provider for your organization and your teams depends on your current IT infrastructure, the business problems you need to solve, and your organization’s business motivations.

Every cloud solution and its implementation should be aligned with the purpose and goal of a business.

Microsoft Azure

Azure is one of the fastest-growing cloud provider platform offered by Microsoft. Even though Azure started years after its competitors, it is one of the leading global cloud computing providers.

Azure offers a wide variety of cloud services in different categories, including Artificial Intelligence, Machine Learning, Analytics, Blockchain, Compute, Containers, Serverless Computing, Databases, Developer Tools, DevOps, Identity Management, Integration, Internet of Things (IoT), Edge Computing, Quantum Computing solutions, Cloud Management and Governance, Media and Communication Services, Azure Hybrid, Migration, Mixed Reality, Mobile, Networking, Security, Storage, Web, and Windows Virtual Desktop.

What makes Azure one of the most attractive and intelligent solutions is its exclusive offering of Microsoft’s products and integration of services in the cloud. Azure pro‐ vides the most advanced and maximum number of smart products and services.

This book will cover Microsoft Azure in detail.

Amazon (AWS)

AWS is Amazon’s cloud platform that offers various services. Some of these cloud services on AWS include Virtual Private Cloud, EC2, AWS Data Transfer, Simple Storage Service, DynamoDB, Elastic Compute Cloud, AWS Key Management Service, AmazonCloudWatch, Simple Notification Service, Relational Database Service, Route 53, Simple Queue Service, CloudTrail, and Simple Email Service.

It is one of the broadly adopted cloud platforms. This cloud platform offer services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

Google Cloud Platform

Google Cloud Platform (GCP) is Google’s cloud and is also one of the top public cloud providers available. Similar to AWS and Microsoft Azure, GCP also offers services in various categories, including computing, storage, identity, security, database, AI and machine learning, virtualization, DevOps, and more. Google Cloud Services are available in 20 regions, 61 zones, and 200+ countries.

GCP is a platform that delivers a wide variety of IT products that IT professionals, businesses, and software developers can take advantage of to work more efficiently and gain more flexibility.

Oracle Cloud

Oracle cloud platform is the cloud offering of Oracle corporation. Oracle Cloud offers IaaS, PaaS, SaaS, and Data as a Service (DaaS). Oracle SaaS offerings are CX, HCM, ERP, SCM, EPM, IoT, Analytics, Data, and Blockchain Applications. Oracle DaaS is the Oracle Data Cloud.

Alibaba Cloud

Alibaba Cloud, founded in 2009, is rarely heard in some parts of the world but is also a prominent public cloud provider. Alibaba Cloud is the largest cloud provider in China. Alibaba is registered and headquartered in Singapore, and it was initially built to serve Alibaba’s e-commerce ecosystem. Nowadays, they also offer public cloud services.

They offer various products and services in multiple categories, including Elastic Computing, Storage and CDN, Networking, Database Services, Security, Monitoring and Management, Domains and Websites, Analytics and Data Technology, Application Services, Media Services, Middleware, Cloud Communication, Apsara Stack, and Internet of Things.

Overall, many cloud computing vendors are coming, but the ones mentioned are widely recognized as top providers. If you want to learn more about how these different public cloud providers are leading, I recommend Gartner Research: Magic Quadrant for Cloud Infrastructure and Platform Services.

Cloud Computing Service Models

There are different cloud service models in the cloud. These are the infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS), as shown in Figure 1-13.

Figure 1-13. The Different Cloud Computing Models show the different scopes of what is managed by the cloud provider and what is managed by the cloud user or organization

Infrastructure as a Service (IaaS)

Infrastructure as a Service (IaaS) is a type of computing model where cloud provider like Azure provides infrastructure as a service. Instead of the traditional infrastructure on-premises in one physical location, they are provided by public cloud vendors.

Infrastructure like web servers, database servers, storage, networking, and computing hardware are available on demand. IaaS in the cloud allows its users to not worry about the maintenance costs of having infrastructure on-premises. The cloud provider controls maintenance, monitoring, security, and load balancing.

Platform as a Service (PaaS)

Platform-as-a-Service (PaaS) is a cloud computing service that allows developers to build and deploy applications on a platform provided by a third-party provider. PaaS providers typically offer a range of tools and services that make it easy for developers to develop, test, deploy, and manage their applications without worrying about the underlying infrastructure.

In a PaaS model, the cloud provider takes care of the hardware, operating system, middleware, and other resources required to run and manage the application. The developer is only responsible for writing the application code and deploying it on the platform. Cloud providers that offer PaaS typically offer various services, including database management, application hosting, security, scaling, and monitoring.

Software as a Service (SaaS)

SaaS allows users to access and use software applications over the Internet rather than installing and running the applications on their computers or servers.

In a SaaS model, the software applications are hosted and maintained by a third-party provider, who makes them available to users over the Internet. Users can access the applications through a web browser or other client software and typically pay for the service on a subscription basis.

SaaS applications can be used for various purposes, including productivity tools, customer relationship management, enterprise resource planning, and many others. Some famous examples of SaaS applications include Microsoft Office 365, Salesforce, and Google Workspace.

SaaS is a software on-demand cloud model where the cloud service providers give the users access to a fully developed application explicitly created for distribution. The software updates are rolled out for all users, and organizations can use their tools with vendor-provided application programming interfaces (APIs).

If we were to compare these main tree the cloud computing service models in the real world, we can look one of our favorite foods, pizza.

Figure 1-14. An example of cloud computing as Pizza as a Service illustration, Image Source: Pizza as a Service

In the pizza illustration example on Figure 1-14 which was originally created in year 2014 by Albert Barron, who used to work as a Software Client Architect at IBM, ⁴.

We can find the similarity between baking our Pizza and having our traditional on-premises IT infrastructure. Shopping for a ready pizza from the store and baking it at home is like infrastructure as a Service (IaaS). Pizza delivery is like Platform as a Service (PaaS), and dining out at a restaurant to order Pizza is like the Software as a Service (SaaS) offered by the cloud provider.

Aside from IaaS, PaaS, and SaaS, other cloud computing service models are available. These additional service models are Serverless, Function as a Service (FaaS), Backend as a Service (BaaS), and more. It will not be a surprise that someday there will be anything X as a Service available for us.

Serverless Computing - Function as a Service and Backend as a Service

Serverless, FaaS, and BaaS are some terms that gained popularity and interest in the cloud computing trends. Serverless is a method of computing where backend services are provided a cloud service by a cloud service provider.

Serverless computing is a cloud computing model where a third-party provider manages the infrastructure and automatically provisions and scales resources as needed, allowing developers to focus solely on writing code for their applications.

In a serverless computing model, the provider handles the server infrastructure, operating system, and other lower-level components while developers provide the application code. This approach can significantly simplify the development process by allowing developers to focus on application logic and functionality without the need to manage the underlying infrastructure.

Serverless computing is related to Backend as a Service (BaaS) and Function as a Service (FaaS). They all provide an abstraction layer between the developer and the underlying infrastructure. However, they differ regarding the level of abstraction and the services provided.

Technically, it is not serverless, but “less” in this term means the servers and underlying infrastructure are abstracted. There are servers behind a serverless function or serverless cloud service, only that the cloud provider or serverless provider is taking it off it for its users. Cloud services on serverless usually have consumption pricing models where the users are only charged on the usage and execution.

Figure 1-15. Example of a Serverless Architecture in Microsoft Azure

Function as a Service (FaaS) is a technical concept that aims to allow developers the freedom and productivity to easily create functions in a cloud environment, as shown in a serverless architecture example in Figure 1-15.

In this method, the developers will still create the application logic, yet the code is executed in stateless compute instances managed by the cloud provider. FaaS provides an event-driven computing architecture where a specific event, such as message queues, HTTP requests, etc., can trigger a function.

In Azure, different serverless solutions, such as Azure serverless compute services for applications and serverless containers, are available.

Figure 1-16. Microsoft Azure Functions as Function as a Service (FaaS)

Azure compute services like Azure Functions, as shown in Figure 1-16, allow users to build applications faster by eliminating the hassles of managing servers and infrastructure. It enables software developers or programmers to focus on the productivity of their development teams and only pay when the code runs. Developers can focus on developing event-driven applications using their chosen supported language.

Implementing serverless and FaaS solutions have several benefits, especially for the software development team. In software engineering, we developers want to focus on delivering solutions, solving problems, and building applications.

Developers and engineers do not want to spend time maintaining the servers and infrastructures. Usually, they prefer to deliver value and solutions by programming and developing applications regardless if they are on-premises or on the cloud.

With serverless computing, FaaS or Backend as a Service (BaaS), developers or cloud engineers can focus more on productivity by focusing on the backend logic and not worrying about infrastructure management. This creates the results of speed of delivery, which helps the project process.

Aside from developer benefits, the opportunity of solving problems with complex applications can be solved. Organizations can also benefit from the speed of delivery and productivity. Other benefits include automatic scaling, reliability, and a consumption-based pricing model.

Serverless architecture is a crucial software architecture design pattern that partly relates to distributed computing systems and microservices. Developers need not worry about managing and maintaining infrastructure, hardware, or servers but can focus more on developing the logic and the functionality. They write the code and leverage the infrastructure of cloud provider services and other third-party services, primarily called BaaS (Backend as a Service). BaaS is a cloud service that takes care of the cloud infrastructure and mainly focuses on the automation of backend-side development.

In Chapter 3, serverless and compute solutions in Azure will be covered in details.

Containers as a Service

n container development, significant features of it come up. Containers as a Service (CaaS) or container development is an exciting solution. By utilizing containers, you get Platform as a Service (PaaS) benefits without the overhead of infrastructure as a Service (IaaS).

Containerization, in simple terms, is deploying your applications into the container. A container is a runtime that contains the essential computing resources needed to run an application. The core part of the host operating system (also known as Kernel) and its shared resources like storage across a host. The shared kernel allows containers to be lightweight and faster.

When hosts are running, the containers in them can be quickly started. Quickstarts mean high availability and resiliency of the applications in the container. One example of containerization technology is Docker, one of the more popular providers of container services. Let us take a look at their different advantages compared to traditional virtual machines.

Containers can run in cross-platform environments

Containers are lightweight and portable

Containers are self-contained and no need to install application dependencies

Containers have good scalability and high availability

Containers are quick to restart

Container development and other compute services in Azure will be covered in the Chapter 3

Data as a Service(DaaS)

Every website, application, system, mobile app, and anything involving tech products we use has data in it. Sensitive data are being protected through data protection policies because data in this modern era is a treasured gem.

Data as a Service (DaaS) focuses on providing data as a business asset by implementing data management strategies. This cloud service model gives organizations better agility in their business.

As listed on Table 1-1, DaaS provides organizations modern, effective and smart strategies on how to handle, manage and visualize huge and massive data that is generated everyday.

Table 1-1. Benefits of Data as a Service (DaaS) as a Cloud Service Model

Benefit
How?

Data-Driven Culture

DaaS enables organizations to organize and manage their increasing data by using datasets that are reusable, easier to analyze, and visualize

Innovation and Business Growth

DaaS puts data as a critical driver in the business, opening growth and innovation opportunities. Data-driven strategies drive innovation and growth without creating huge risks.

Scalability, Reliability and Flexibility

Cloud solutions usually offer DaaS solutions that are flexible and scalable

Data Monetization

Solving data operation problems and complexity can also be beneficial in monetizing the valuable data

Cost Savings

DaaS solutions can help organizations save costs of expenses by allocating the appropriate workloads for their data in the cloud

Maintenance Tasks Automation

DaaS solutions can help organizations save costs of expenses by allocating the appropriate workloads for their data in the cloud

The Challenges of Data as a Service (DaaS)

Although Data as Service (DaaS) offers great benefits to an organization, it has some known challenges.

Risks of solving data complexity problems especially for old and unstructured data sets

Implementing a data-driven culture with DaaS require an top-down organizational and business strategy

Higher demand for management of data privacy and security because of the different data privacy regulations and compliances

Shared Responsibility in Cloud Computing and Azure

Considering and evaluating cloud services on any public cloud provider require careful planning and strategy for an organization. It is critical to learn and under‐ stand the shared responsibility model. The shared responsibility model helps both parties (the user and cloud vendor) share the trust and responsibility of hosting applications and resources in the cloud.

The Shared Responsibility Model illustrates shared trust and responsibility that helps in cloud security and the protection of resources. For example, it helps identify which security tasks are handled by the cloud provider and which jobs are handled by the public cloud user. The shared responsibility for the workload varies depending on whether the workload is hosted on Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), or in an on-premises data center.

With Microsoft’s example of the Cloud-Enabled Shared Responsibility Model Considering and evaluating cloud services on any public cloud provider require careful planning and strategy for an organization. It is critical to learn and under‐ stand the shared responsibility model. The shared responsibility model helps both parties (the user and cloud vendor) share the trust and responsibility of hosting applications and resources in the cloud.

The Shared Responsibility Model illustrates shared trust and responsibility that helps in cloud security and the protection of resources. For example, it helps identify which security tasks are handled by the cloud provider and which jobs are handled by the public cloud user. The shared responsibility for the workload varies depending on whether the workload is hosted on Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), or in an on-premises data center.

Shared Responsibility Model offers Cloud Security Advantages

The cloud offers significant advantages for solving long-standing information security challenges. In an on-premises environment, organizations likely have unmet responsibilities and limited resources available to invest in security, creating an environment where attackers can exploit vulnerabilities at all layers.

The shared responsibility model is a security framework that outlines the security responsibilities of both cloud service providers and their customers. This model is commonly used in cloud computing, where customers use cloud services provided by a third-party provider to store, process, and manage their data.

In this model, the cloud service provider, such as Azure, is responsible for the security of the cloud infrastructure, including the physical data centers, networks, and hardware. This includes ensuring the infrastructure’s availability, integrity, and confidentiality, as well as maintaining compliance with relevant laws and regulations.

On the other hand, the customer is responsible for securing their data and applications stored and used in the cloud. This includes implementing appropriate access controls, managing user identities and authentication, and securing data in transit and at rest.

In essence, the cloud service provider and the customer have a shared responsibility for security, with each party responsible for different aspects of the cloud environment’s security. The cloud provider and the company or user must understand their responsibilities to ensure the cloud environment is secure and compliant with relevant regulations.

Figure 1-17. Example of Microsoft’s Cloud Enabled Shared Responsibility Model (Image Credit: Microsoft Documentation, “Shared responsibility in the cloud”)

Figure 1-17 shows a traditional approach where many security responsibilities are unmet due to limited resources. In the cloud-enabled practice, you can shift day-to-day security responsibilities to your cloud provider and reallocate your resources.

Capital Expeditures and Operational Expeditures

An organization can’t migrate on-premises resources to the cloud without knowing about its benefits of it for Capital Expenditures (CapEx) and Operational Expenditures (OpEx). Identifying the benefits of comparing these two when considering cloud computing solutions for businesses or organizations is vital.

Capital Expenditure (CapEx)

On the other hand, OpEx refers to expenses that are incurred on an ongoing basis to keep the business running, such as cloud computing services, software licenses, salaries, and maintenance costs. These expenses are typically variable and can be adjusted depending on the level of usage, making them more flexible than CapEx. With the adoption of cloud computing, businesses can reduce their CapEx and shift toward OpEx, as they can pay for the services they use on a pay-as-you-go basis, allowing for more efficient resource allocation and cost savings.

Operational Expenditure (OpEx)

On the other hand, OpEx refers to expenses that are incurred on an ongoing basis to keep the business running, such as cloud computing services, software licenses, salaries, and maintenance costs. These expenses are typically variable and can be adjusted depending on the level of usage, making them more flexible than CapEx. With the adoption of cloud computing, businesses can reduce their CapEx and shift toward OpEx, as they can pay for the services they use on a pay-as-you-go basis, allowing for more efficient resource allocation and cost savings.

Typically, organizations start with traditional on-premises physical servers and data centers that require expensive CapEx. Expenses for data center physical space, hardware, equipment, and employees to manage everything in place - security, maintenance, etc. can be very costly.

Cloud computing solutions offer organizations the option to eliminate the hassles of traditional infrastructure on-premises by providing services with OpEx alternatives.

Benefits of Adopting and Transformation to Modern Cloud Technologies

Cloud adaptation and transformation is a complex and lengthy process. It is not a quick-fix solution to modernize on-promise and legacy applications quickly to the cloud. However, when planned adequately with intelligent strategies, there is an excellent list of benefits.

Cloud Computing for Business Value and Customers

How does an organization or business benefit from cloud computing? Regardless of the size of the business or organization, cloud computing helps in saving resources, time, and finances by accelerating innovation, collaboration, modernization, and productivity in different teams within the organization. It also helps provide business value for users because of the enhanced user experience, speed, and reliability of modern cloud applications.

Cloud Computing for IT Companies

IT Companies and organizations gain many benefits from using cloud computing solutions.

Data Access Management and Portability

Cloud computing lets businesses portably access their important business-related data anywhere on any device. This capability allows the entire organization to work effectively and productively by focusing on deliverables. With cloud storage and servers on the cloud, employees in the organization can be out of the office intranet to work and collaborate. Instead, information is available securely at any time on demand.

Cost management and efficiency

Buying and maintaining server equipment requires time, expertise, and money; rather than building your bespoke server, which can be prone to downtime, a cloud computing provider stores data for you without all the downsides. Prices for business-oriented cloud services are still monthly, but it’s a manageable and predictable expense in many aspects.

Convenient Backup and Disaster Recovery Solutions

Catastrophic data loss can happen at any time, and it can be time-consuming to solve such a significant issue when it happens. Whether that loss occurs from natural disasters, power surges, or hardware failure, affected companies are at increased risk of bankruptcy within the same year as the data loss. And while most companies have adopted backup plans, it helps to have additional contingencies. By utilizing the cloud to store important data, business owners can rest easy knowing that important files are safe even if hardware fails. For cloud computing solutions in Microsoft, there are a great variety of backup and disaster recovery options available for applications hosted in Azure.⁵

High-Level Cloud Security and Data Privacy

Security and privacy are critical in the decision-making process when using cloud computing services. The public cloud computing provider and the user or organization have shared responsibility. Hosting applications and servers on the cloud are built on trust. This is the reason for setting clear expectations and being familiar with the shared responsibility model. Cloud providers prioritize security and data privacy for their clients and consumers. They use different strategic cloud security controls to protect their users’ resources. Identity management, high-level physical security in data centers, strict personal protection, and ensuring data privacy for sensitive data are taken seriously. Dedicated security expert teams scan the cloud for possible internal and external vulnerabilities. Performing cloud penetration testing and examining inside and outside the cloud must be strictly authorized. Cloud providers do not tolerate any security breaches. Termination of services in case of violation of security and use policies are implemented. It may not be noticeable to many, but Cloud computing is advancing our society worldwide. It also impacts our daily routines, work, home, and school in so many ways. It has improved our facilities for healthcare, education, and communities, and it is continually changing for our digitalization worldwide.

Digitalization and Modernization

Digitalization and modernization of many computer systems in different sectors were made possible because of the advancement of our technologies hosted on the cloud over the Internet. Institutions in other sectors like healthcare, governments, community, schools, and organizations are using the cloud to modernize.Their services and products. The ability to make things more accessible through technology is an advantage.

Remote and Flexible Education and Digital Literacy

Aside from digitalization and modernization in different sectors of our society that benefits us widely, cloud computing also helped to get better education worldwide. With the power of the Internet, cloud computing enables us to work from home or anywhere. It activates the capability and capacity of online collaborative and self-paced modern learning environments.⁶. Remote locations with internet access facilities give opportunities for literacy through online education.

Summary

In this chapter, you have learned about the fundamentals of cloud computing, its different types, and its deployment models that give you an understanding of the cloud. You also know about the history and evolution of cloud computing that helped shape the modern cloud that allows our society today.

You saw an overview of the different types of clouds public, private, hybrid, community, and multi-cloud, and know the differences between them. The other deployment models, such as IaaS, PaaS, SaaS, CaaS, DaaS, FaaS, and Serverless, are helpful when choosing the exemplary cloud service to use in a cloud platform.

You got insights and learned about the advantages and benefits of cloud computing in businesses, IT companies, our society worldwide, and software engineering.

In the next chapter, we will learn more about the vital concepts you need to know about Microsoft Azure as a cloud platform. We will learn what cloud solutions Azure has to offer to help you and your organization.

Check Your Knowledge

What is cloud computing?

What is the difference between private cloud and public cloud?

Why is it important to understand the shared responsibility model in the cloud?

What is the different between hybrid cloud and multi-cloud?

What is CapEx and OpEx?

To find the answers to the Check Your Knowledge, turn to back of the book in the Appendix

Recommended Resources

Dataversity.net, A Brief History of Cloud Computing by Keith D. Foote

Mainframes and Cloud Computing: Similarities and Differences

Microsoft Azure - What is the cloud?

MIT Technology Review - Who Coined Cloud Computing? by Antonio Regalado

Cloud Computing Market by Service, Deployment Model, Organization Size, Vertical And Region - Global Forecast to 2026

What is Cloud Computing? A Beginner’s Guide

Microsoft’s Documentation - Cloud Computing Basics for Developers

Gartner.com - Magic Quadrant for Cloud Infrastructure and Platform Services

¹ Keif Morris, 2020, “Infrastructure as Code”, O’Reilly Media, https://learning.oreilly.com/library/view/infrastructure-as-code/9781098114664

² Bernard Marr, 2021, “The 5 Biggest Cloud Computing Trends In 2022”, Forbes.com, https://www.forbes.com/sites/bernardmarr/2021/10/25/the-5-biggest-cloud-computing-trends-in-2022/

³ Shardul Bhatt, 2021, Using Cloud Computing to Build a Remote Organization”, Business2Community.com, https://www.business2community.com/cloud-computing/using-cloud-computing-to-build-a-remote-organization-02382933

⁴ Pizza as a Service, “Pizza as a Service” by Albert Barron, https://www.linkedin.com/pulse/20140730172610-9679881-pizza-as-a-service/

⁵ Microsoft Documentation, “Backup and disaster recovery for Azure applications”, https://docs.microsoft.com/en-us/azure/architecture/framework/resiliency/backup-and-recovery

⁶ The Computer Society, “Cloud Technologies in the Education System"by James Riddle, https://www.computer.org/publications/tech-news/build-your-career/cloud-technologies-in-the-education-system

Chapter 2. Microsoft Azure Fundamentals

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at jleonard@oreilly.com.

Cloud computing has revolutionized the way businesses run their IT workloads. It allows organizations to not only build and deploy applications in a modern way by using modern PaaS/SaaS services but also makes use of cloud computing advantages like high availability, scalability, security, and cost optimization. Understanding the Fundamentals is critical to building an excellent foundational knowledge of Microsoft Azure that later you can use to build and manage complex workloads. If you are just getting started in your cloud career or if you need to get deeper knowledge on Microsoft Azure, get your fundamentals right!

Tiago Costa, Cloud Architect and Advisor, Microsoft Azure MVP, Microsoft Certified Trainer

Previously, we learned about the important concepts you need to know about Cloud Computing, which helps us and many organizations use and avail cloud computing resources on flexible pricing models with support for global scale, flexibility, reliability and security.

In this chapter, you enhance your knowledge of cloud computing by digging deeper into the world of Microsoft’s cloud computing platform, Azure. You will learn about the core concepts of what Azure is as a cloud provider. You learn how Azure started, what is its purpose and how it can inspire you to be innovative as you build modern solutions for your organization’s demands and requirements.

You will learn about Azure Geographies, Regions and Availability Zones as well as Azure Resource Manager which gives you an overview of Azure’s cloud infrastructure.

This chapter will give you a comprehensive overview of the core services in Azure. Each of those core Azure services will be explained in detail with its uses and examples in the upcoming chapters of this book.

Note

The Public cloud cloud is a type of deployment model where the cloud infrastructure is available to the public or any organization, and the resources are shared with other organizations or other cloud provider tenants.

Microsoft Azure as a Public Cloud Provider

Microsoft’s computing platform is called Microsoft Azure, also known as Azure. The term Azure means sky blue color and is usually the typical color of “the cloud. " Historically, the cloud platform was called Windows Azure that was originally built on Windows Server 2008 for developers who want to host their software applications like Windows environment to the cloud. footnote:https://docs.microsoft.com/en-us/archive/blogs/aniyer/windows-azure-microsofts-cloud-platform[Microsoft Documentation, “Windows Azure - Microsoft’s Cloud Platform]

Note

Windows Azure became Microsoft’s foundation of cloud and was commercially available to the public. Windows Azure became Microsoft’s great and powerful cloud platform. In 2014, it was renamed to Microsoft Azure. Originally, Azure was a project with a code name Red Dog.

Azure’s flexibility of being able to create, build, deploy, and manage organizations’ applications on a global scale made it one of the top cloud providers worldwide. Microsoft Azure, grew to become a public cloud provider used by 95% of Fortune 500 companies.

Microsoft Azure Helps Organizations with Minimizing Upfront Costs

The public cloud has features that are useful to many. These features include not requiring upfront costs for Capital Expenditures (CapEx) to scale up resources, quick provisioning and de-provisioning of applications, and flexibility for organizations to only pay for what they use.

Azure is a public cloud provider, but they also offer private, hybrid, and multi-cloud solutions to its users. Cloud services in Azure are designed to help its users build new and innovative cloud solutions that help solve our challenges in different areas.

It allows us to build, develop, manage, and run resources like servers, databases, storage, or applications in multiple cloud environments. You can use it for different use-case scenarios in the cloud using the tools, programming languages, and frameworks that you prefer to use.

Azure offers over 200 services¹ in various categories, including computing, networking, storage, databases, analytics, machine learning and AI, the Internet of Things (IoT), cloud-native, containers, and security. However, Microsoft continues to introduce new features and services to Azure regularly, so the number of services may increase over time.

Azure also provides edge computing, AI, and Machine Learning (ML) services that enable you to create smart and intelligent solutions with your existing or new applications. Without the technology of cloud computing, this would not be possible.

Benefits of a Cloud Provider

There are several benefits that a cloud provider has over the traditional on-premise physical environment, as shown on Table 2-1.

Table 2-1. The list of known benefits

Benefit
Description

High availability

Microsoft Azure provides high availability and redundancy across all of its worldwide data centers offering a service-level agreement that ensures 99.95% availability

Geo-distribution

Azure helps in the global enterprises by providing geo-distribution features. Geography-specific endpoints enables international enterprises comply to the regional compliance and regulations

Scalability On-demand

When demand for complexity, traffic and data expands, there should be a flexible and quick way to handle such need

Reliability

The system or application hosted should function correctly even in the face of adversity (hardware or software faults, and even human error)

Elasticity

A capability to automatically scale cloud resources based on configuration or demand

Disaster recovery

When your applications, data and systems are hosted in Azure, you can be assured of secured end-to-end backup and disaster recovery solution

Flexibility

Cloud services in Azure gives organizations flexibility by allowing them to use consumption pricing plans and full self-service management accessible anywhere

Cost Management Tools

There are tools available for cost management in Azure and users can also set budget alerts for their resource groups and resources

Advanced Compliance

Azure takes information security and compliance seriously with high standards to serve best its users

High Level Cloud Security

Azure protect all stored data with world-class and advance security. Data stored in Azure are protected with advanced encryption technologies. All data centers of Microsoft are equipped with extreme security with biometric scanners, multi-level authentication and more

OpEx vs. CapEx

Hosting to cloud means an organization can save money from Capital Expenditures (CapEx) and only pay for Operational Expenditures (OpEx)

Consumption-based pricing model

Azure gives its users the flexibility of cost management by offering consumption-based (Pay-As-You-Go) pricing model in most of its cloud services.

No deep technical skills required

You and your organization do not need to be very skilled technically to get started using the cloud platform. Azure provide a flexible and diverse options to give ease of use for its users

Azure Portal

The self-managed portal of Microsoft’s cloud platform is called the Azure Portal, which can be accessed by Azure users on their web browsers or via the Azure Mobile App. It is a web-based administration website for all types of Azure users. It is where you can manage your Azure cloud services for your organization. It is a powerful portal of cloud administration tools and resources you need. For example, in Azure Portal, you can manage your resource groups, resources in it, Azure subscriptions, security, monitoring and more, as shown on Figure 2-1. It has all the services available that you can use for your cloud computing solutions.

Figure 2-1. Example homepage of Azure Portal accessed on a web browser

Tip

Access Microsoft Azure Portal using Azure Mobile App. You may download it using the instructions on the link Get Azure Mobile App

Access Microsoft Azure Portal using your favorite web browser using the link https://portal.azure.com/

The Azure Portal is built to be resilient, portable and accessible from anywhere. Figure 2-2 shows the user interface of the Azure Mobile App which you can use to manage your Azure subscriptions and services on the go.

Figure 2-2. Start page of Azure Mobile App (iOS or iPhone)

Tip

The website URL of the Azure Portal is specific to the cloud where your organization is deployed in Microsoft Azure.

For commercial use or Azure Public Cloud: https://portal.azure.com

For Azure United States Government Cloud: https://portal.azure.us

Azure Germany: https://portal.microsoftazure.de

Azure China: https://portal.azure.cn

Aside from the different locations and different scope of Azure as a cloud platform in government sector and countries, there is also Microsoft Cloud for Sovereignty. It is a solution that will help Azure users in government and public sectors in the leverage the cloud based on their specific and unique requirements. These requirements can be their compliance, security and policy requirements. It will help these sectors gain better control over their data, privacy, compliance, governmance, etc.

Features of Azure Portal

The Azure Portal has numerous of features for all types of Azure users. Organization can take control of cloud resources by governing their cloud resources on-demand globally. Software developers or cloud engineer can build, manage and monitor any type of cloud applications - from simple to complex ones regardless of architecture or programming languages.The following are some of the known features of the Azure Portal.

Create, build, manage and monitor Azure services and cloud resources all in one place at anytime and anywhere at your own convenience

Use command line tools and cloud shell for quick creation and deployments

Manage and organize Azure Subscriptions and create management groups that helps in structuring and governing Azure resources

Azure Active Directory (Azure AD) as a great tool for the management of identity, access, and permissions to resources

Configure and manage privacy, data, security, policies and compliance which is vital for the organization’s governance

Customization of portal’s dashboards helps in getting a quick overview of the status of resources right after you logged in

Take control of monthly costs by monitoring resources through spending limits and budget alerts using using Azure Cost Management in the Azure Portal

Search everything you need to know, create, build and manage using the Global Search feature in the portal

Send Azure Support requests directly when you need assistance or help

Tip

Azure Marketplace is a marketplace for Azure customers to search, purchase and try-out applications and services from other service providers including Microsoft partner companies. All services on Azure Marketplaces are verified and certified to work with the Azure cloud platform.

Microsoft Azure Services

As of writing this book, all Azure services are divided into 21 categories according to their purpose. As a developer or a solution architect in an organization, or even as a beginner, it might be overwhelming to see a lot of cloud services in a cloud platform such as Azure. However, each Azure service has its unique purpose and is built to solve specific technical problems. Azure services can be seamlessly integrated with other services and even external ones.

Each category for the Azure services helps you build and integrate cloud solutions based on your business requirements or need. For example, building a web application with integrations of API, cognitive services and reporting features would need several Azure services in categories Artificial Intelligence/Machine Learning, Compute, Analytics, Databases, Integrations, Developer Tools, etc.

Table 2-2. Overview of Microsoft Azure services in some known categories

Category
What are these categories?
Azure services

Artificial Intelligence (AI) + Machine Learning (ML)

Build the modern cloud apps with ML and cognitive integration

Azure Boot Service, Azure Cognitive Services Azure Machine Learning, Anomaly Detector, Azure DataBricks, Azure Open Datasets, Computer Vision , Face API, Azure Immersive Reader, Azure Form Recognizer, Kinect DK, Microsoft Genomics, Azure Health Boot, Azure Applied AI Services, Azure Percept, Speech Services, etc.

Analytics

Gather and visualize any time of data regardless of its volume or velocity

Azure Analysis Services, Azure Data Explorer, Azure Data Lake Storage, Azure Data Share, Azure Databricks, Azure Stream Analytics, HDInsight, PowerBI Embedded, Azure Synapse Analytics, Data Factory, Event Hubs, R Server for HDInsight, Microsoft Graph Data Connect, Azure Purview, etc.

Compute

Build robust applications with high-level cloud compute capacity and features of scalability

API Apps, App Service, Azure Cycle Cloud, Azure Functions, Azure Kubernetes Service (AKS), Azure Quantum Preview, Azure Spot Virtual Machines, Azure Spring Cloud, Azure VMware Solution, Azure Batch, Cloud Services, Linux Virtual Machines, Azure Container Instances, Azure Static Web Apps, VM Scale Sets, Azure Virtual Machines, Azure Virtual Desktop, Web Apps, Azure Dedicated Host, Azure VM Image Builder, etc.

Containers

Create, build, develop and manage containerized applications with modern integration tools

Azure Kubernetes Services (AKS), Azure Container Instances, Azure Container Registry, Azure Service Fabric, Web App for Containers

Databases

Fully managed and secure cloud database services

Azure SQL Database, Azure Cosmos DB, Azure Cache for Redis, Azure Database for PostgresSQL, Azure Database for MySQL, Apache Cassandra MI, SQL Server on Virtual Machines, Azure Database Migration Service, Table Storage, Azure API for FHIR, Azure SQL Database Edge, etc.

Developer Tools

Services and development tools for engineers or developers working with cloud development and DevOps for Azure

Azure DevOps, Azure DevTest Labs, App Configuration, Azure SDKs, Azure Lab Services, Azure Pipelines, Visual Studio, Visual Studio Code

Integration

This is a category of services on Azure for different types of integration within Azure, hybrid or multi-cloud

API Management, Azure Event Grid, Azure Service Bus, Azure Logic Apps, Azure Web PubSub Preview, Azure Healthcare APIs Preview

Networking

Connect cloud and on-premises infrastructure and resources using networking services

Application Gateway, Azure Bastion, Azure DNS, Azure Express Route, Azure Content Delivery Network, Load Balancer, Azure Front Door, Azure Firewall, Internet Analyzer, Azure Orbital, Private Link, VPN Gateway, Virtual WAN, Virtual Network, Traffic Manager

Internet of Things (IoT)

Create cloud solutions with IoT services

Azure IoT Hub, Azure IoT Central, Azure Sphere, Azure IoT Edge, Azure RTOS

Identity + Security

Protect resources, data and identity on the cloud

Azure Active Directory, Azure AD B2C, Azure Defender, Azure Security Center, Azure Key Vault, Azure Sentinel, Information Protection, DDoS Protection, etc.

Table 2-2, describes some of the common known Azure services in different categories. To view the updated list of services per category, please visit website https://azure.microsoft.com/en-us/services/

Some of the important Azure services in several categories will be covered in the other chapters of this book.

Overview of the Core Services of Microsoft Azure

Microsoft consistently creates new solutions, continuously updates, and improve Azure services all the time.

Compute Services in Azure

Compute is usually the term used for computing resources. Compute services hosted in Azure provides computing resources like the operating systems, networking, disks, processors, networking, and memory. These compute resources are available quickly and on-demand by its users. Every application is unique. An application can have many workloads that need more than one compute services.

As of date of writing, Azure has about more than 25 compute services available. The Azure Compute services in Azure enable us to build web and mobile applications, deploy and manage virtual machines, build apps in containers in the cloud, create batch jobs, and more. Figure 2-3 shows an overview of compute services on Azure Portal. Additionally, Table 2-3 gives us an overview of some of the common Azure Compute services and their purpose.

Figure 2-3. An overview of Compute services in Microsoft Azure Portal

Table 2-3. Overview of Azure Compute Services

Azure Compute
What is it for?

Azure App Service

Build and develop web and mobile apps in a fully managed cloud environment

Azure Static Web Apps

Develop modern full stack web application quickly to Azure from a code repository

Azure Virtual Machines

Quick, easy and manageable provisioning of Virtual Machines (Azure VMs) in different operating systems like Windows or Linux

Azure Virtual Machine Scale Sets

Create and provision multiple and thousands of Virtual Machines(Azure VMs) with high availability advantage

Azure Spot Virtual Machines

Save money when you provision compute capacity you don’t use for your workloads

Azure Functions

Develop serverless, modern, event-driven applications, and stateful workflows

Azure Container Apps

Build and deploy fully-managed modern apps and microservices using serverless containers

Azure Kubernetes Service(AKS)

Build managed Kubernetes containers on the cloud

Service Fabric

Build microservices and perform containers orchestration in different operating systems like Windows and Linux

The technical details, use cases, and how you can get started developing with Azure compute services are covered in Chapter 3 Microsoft Azure Cloud Compute

Networking Services in Azure

Networking in the cloud is important in all aspects of cloud hosting. Networking services help secure both private and public cloud infrastructure. Users can customize their cloud networking setup and manage their network resources on demand.

Azure Networking services allow Azure users to meet the demands of their infrastructure’s workloads on-premises, hybrid or on the cloud with high availability and enterprise-level Microsoft Zero Trust based security on networking services.

Table 2-4 shows some of the common Azure Networking services.

Table 2-4. Overview of Azure Networking services

Azure Networking
What is it for?

Azure Virtual Network

Connect virtual machines using VPN connections

Azure Bastion

Secure and easy access to your virtual machines using private RDF and SSH that are fully managed

Azure Private Link

Access cloud Azure-hosted services with privacy

Azure Firewall

Protect your resources in the cloud with high availability and low maintenance firewall

Azure Load Balancer

Load balance your application connections and requests - both inbound and outbound

Azure ExpressRoute

Create private network connections between Azure data centers and on-premises infrastructure

Azure Traffic Manager

Choose to route your network traffic for better performance

Azure VPN Gateway

Create secure private network connections in the cloud VPN

Learn more about the technical details and how you can get started with Azure networking on Chapter 4, Microsoft Azure Cloud Networking

Core Azure Storage Services

The storage services in Azure offers great storage for any types of data objects, Azure Virtual Machine disk storage, reliable messaging storage, and other modern data types that needs to be stored. They provide the benefits of high availability, durability, security, accessibility and manageability.

Table 2-5 shows some of the common Azure Storage services.

Table 2-5. Overview of Azure Storage services

Azure Storage service
What is it for?

Azure Blobs

Store scalable binary data, text or Data Lake Storage Gen2 big data analytics

Azure Files

Fully manageable file shares for deployments on-premises or for the cloud. Accessible anywhere through Server Message Block (SMB) Protocol

Azure Queues

Using Queues you can collect large messages that you access via authenticated HTTP calls

Azure Managed Disks

Store block-level volumes for Azure Virtual Machines

Each of the core Azure storage service need to be integrated and associated with an Azure Storage account. A storage account in Azure is a container, as shown on Figure 2-4, of all Azure storage data like the blobs, files, tables and queues.

Figure 2-4. An Azure Storage Account is needed for any of the core Azure storage services

Core Azure Database Services

Azure has a great collection of database services to select from depending on the type of data structure you need for your organization. The different types of database services in Azure are fully managed. There are appropriate databases for your data - relational, NoSQL, in-memory databases, and other modern databases.

Table 2-6 shows some of the common Azure Database services.

Table 2-6. Overview of Azure Database services

Azure Database service
What is it for?

Azure SQL Database

Cloud-hosted SQL databases that are fully-managed, intelligent and secure

Azure Cosmos DB

Create and migrate no NoSQL workloads to the cloud like Cassandra, MongoDB, and other NoSQL databases

Azure Cache for Redis DB

Build fast and scalable applications with Redis in-memory data store

Azure Database for PostgreSQL, MySQL, and MariaDB

Create fully managed and scalable databases for PostgresSQL, MySQL and MariaDB

Azure SQL Edge

Build IoT edge-optimized SQL database engine with built-in AI

Learn more about the other storage and database solutions as well as its technical details on Chapter 5, Microsoft Azure Cloud Storage Services and Databases

Identity Management and Security Services

Secure your organization’s cloud resources against advanced modern threats using the identity management and security services in Azure.

Table 2-7 shows some of the common identity management and security services in Azure.

Table 2-7. Overview of Azure Identity and Security services

Azure Identity or Security service
What is it for?

Azure Active Directory (Azure AD)

Secure identity and protect users of the entire organization using SSO and multi-factor authentication

Azure Information Protection

Protect your sensitive information on the cloud

Azure Key Vault

Allows you to keep control, manage, and secure your keys, connection strings and secrets

Azure Defender

Protect and detect threats for your workloads in Azure, on premises, and even in other cloud providers

Microsoft Sentinel

Gather intelligent security security information event management (SIEM) a and security orchestration automated response (SOAR) b solution to protect your resources

Azure DDoS Protection

Protect applications in Azure from Distributed Denial of Service (DDoS) attacks c

Azure Information Protection

Protect your sensitive information on the cloud

a Wikipedia, “Security information and event management, https://en.wikipedia.org/wiki/Security_information_and_event_management

b Rapid7.com, “Security Orchestration Automation and Response (SOAR) Tools and Solutions, https://www.rapid7.com/solutions/security-orchestration-and-automation/

c Cybersecurity & Infrastructure Security Agency, “Security Tip (ST04-015) - Understanding Denial-of-Service Attacks”, https://www.cisa.gov/uscert/ncas/tips/ST04-015

What is SIEM System?

SEIM is the abbreviation of Security Information and Event Management which is a computer security system that can be used as a tool to collect, analyze, and perform security operations on computer systems, which can be applications or hardware. A SEIM system has features like collection and logging data from resources within your environment, creating alerts for potential security anomalies, incident management and data log visualization.

Learn more about the technical details and how you can get started with Azure user identity platform and security services on Chapter 9 Cloud Security, Compliance and Identity Services in Azure

Developer Tools, Monitoring, and DevOps Services

Azure have different tools and services for cloud development, troubleshooting, monitoring, DevOps practices, Infrastructure as a Code (IaC), and Continuous Integration/Continuous Delivery (CI/CD).

Table 2-8 shows some of the common Azure developer tools and DevOps services.

Table 2-8. Overview of Azure developer tools and DevOps services

Azure service
What is it for?

Azure DevOps

All-in-one tool with great DevOps services for teams to collaborate better, share code, track work, and deliver software projects

Azure DevTestLabs

Quickly create environments using reusable templates and artifacts

App Configuration

Store your application’s configuration using scalable parameters

Visual Studio

Develop, debug, deploy, manage, and diagnose cloud-scale applications on Azure, using a full-featured IDE

Visual Studio Code

Write and debug code with a lightweight and fast code editor that runs on operating systems like Windows, Linux, and other supported operating systems

Other development tools for Azure is discussed on Chapter 14 of this book.

Table 2-9 shows some of the common monitoring services in Azure.

Table 2-9. Overview of useful azure monitoring tools in Azure

Azure service
What is it for?

Azure Monitor

A great tool for maximizing application performance by collecting, analyzing, visualizing and automating telemetry data in Azure and on-premises environments

Application Insights

Provides features that are useful for application performance management (APM) such as live monitoring and automatic detection of performance issues

Azure Advisor

An innovative cloud assistance in Azure that assist you in improving your deployments by recommending useful and actionable solutions that helps in securing resources, save cost, and improving performance

Log Analytics

It allows you to edit, run log queries and analyze the data collected by Azure Monitor Logs

The different core developer tools, supported programming languages for Azure, Monitoring, troubleshooting steps and DevOps services will be discussed in details on Chapter 11, Developing with Microsoft Azure, Troubleshooting, Monitoring, DevOps, Automation, CI/CD

Cloud Migration and Hybrid + Multicloud Cloud Services

Azure has migration and hybrid solutions that helps organizations in their cloud adoption and migration journey.

Table 2-10 shows some of the common Azure migration, hybrid and multicloud services.

Table 2-10. Overview of Azure services for migration, multi-cloud and hybrid cloud

Azure service
What is it for?

Azure Database Migration Service

Get guidance and useful tools for migrating databases from different on-premise resources to Azure

Azure Data Box

Device solution for data transfer of large amounts of data to Azure and edge compute

Azure Migrate

All in one place for all the guides and tools you need to migrate to Azure

Azure Arc

Combine and unify on-premises, hybrid, and multicloud infrastructure

Azure Stack

Build and run innovative hybrid apps across cloud boundaries

Learn more about the technical details and how you can get started with Azure migration and hybrid cloud solutions on Chapter 13, Microsoft Cloud Migration and Hybrid Cloud Architecture Solutions

Those are the core services in Microsoft Azure in different major categories. Each of them along with some of the categories listed in Table 2-2 will be described in details with examples in the upcoming chapters of this book.

Core Architecture and Resource Management Concepts in Microsoft Azure

An organization in Microsoft Azure would need their Azure administrators to properly setup the core structure of the organization in the resource management. This is the first and top structure that is required before you can add cloud resources in the cloud platform. An organization with different departments with members in different roles would find this beneficial.

There are four levels for organizing your organization’s resources in Microsoft Azure. As shown on Figure 2-5, from top to down, these are the Azure Management Groups, Azure Subscriptions, Resource Groups and Resources.

Figure 2-5. The top-down structure of the four levels in organizing Azure resources, Image Credit: Microsoft

Azure Management Groups

Azure Management Groups is the top level of the core structure of managing your cloud resources in Azure. It is useful in organizing and structuring your resources in Microsoft Azure. The management groups is where Azure administrators manage everything about user access, compliance and policies for subscriptions. The subscriptions within a management group automatically inherit the settings, conditions and restrictions added in the group.

Azure Role-based Access Control (RBAC) for all resources and role definitions are supported in the management groups. A person in the organization with any Azure role can be assigned to the Azure management group. Those who have access and rights to the management group can group Azure Subscriptions, see organization’s management groups hierarchy and most of all can control any access to any Azure service or resource by creating and applying governance control and policies.

Azure Subscriptions

Subscriptions in Azure is like a big container for all accounts of users and what resources they have accessed or used within the subscription. Every subscription usually has amount limit of resources that a certain user can create and use. As an organization, you can use subscriptions to control the monthly bill and resources costs in your organization or your own Azure account. Using Azure subscription, the organization can also control what resources the users create, update, or delete.

Azure Resource Groups

Azure users can group their services or resources using Azure Resource Groups. A resource group in Microsoft Azure acts as logical container where resources like servers, web applications, databases, storage, monitoring, etc. are deployed, managed, and stored. Do not confuse a Resource Group for Availability Set in Azure. The Availability Set is the logical group for Virtual Machines (VMs).

Azure Resources

The databases, servers, virtual machines or web applications you create on Azure platform are considered Azure resources. All resources or services you create must be in added in a Resource Group. which acts as the logical container. In a resource group, you can have your web apps, servers, monitoring, compute services, etc. in one place.

When you are creating an Azure resource as shown on Figure 2-6, you would need to select an Azure Subscription, a Resource Group and the Azure Region for example, North Europe, where you want to add your resources.

Figure 2-6. Creating Azure Resource in Azure Portal

Figure 2-7. Using tag to categorize Azure resources

You can also organize your Azure resources in categories by adding tags as shown on Figure 2-7. These resource tags are key-value pair set that are composed of name of your tag and the value. You can set tags for categorizing your resources for billing purposes. For example, you can apply tags for your resources for different environments like resources in Dev, Test, UAT, or Production.

Azure Resource Manager

Azure Resource Manager (ARM) is also an important element in managing resources in Azure. ARM is the management and deployment service that provides users the capability to add, edit and delete resources in Azure. By using ARM, the organization can manage user access control and organize resources securely even after deployment.

Azure Resource Manager templates (ARM templates) are commonly used to automate deployments and to implement Infrastructure as Code (IaC). Azure provides native support for IaC using the ARM templates and also Azure Bicep in Azure Resource Manager.

Terraform is also supported on Azure. IaC creates a great advantage, and it enables good deployment automation of the infrastructure in the cloud. Using Infrastructure as Code, you can automate your deployment by generating templates for the same environment every time. Infrastructure as Code process minimizes the problems of environment drift during development releases.

Azure provide third-party support for other automated IaC platforms like Terraform, Red Hat Ansible, Chef Automate, and Pulumi.

Azure Bicep uses declarative, concise, and type-safe syntax to deploy Azure resources and is considered a domain-specific language (DSL). It promises the best developer experience when it comes to authoring IaC solutions in Azure. We will go through the technical aspects of it in the later chapters of this book.

Azure Geographies, Regions, Region Pairs, and Azure Availability Zones

The Microsoft Azure infrastructure is built to deliver the best resiliency and high availability of cloud resources to its users. Azure infrastructure is composed of its different geographies, regions, and availability zones across the world. Therefore, knowing where to provision your resources in the right location in Azure’s global infrastructure will hugely help in an organization’s cloud resources management, compliance, security, and speed.

Azure Geographies

As of writing this book, Microsoft Azure has data centers worldwide in 140 countries. The data centers of Azure are secured physical buildings that are located all over the world. The entire global infrastructure of Azure is huge and widely available globally. The data centers and infrastructure of Microsoft’s cloud platform are important in providing reliable, secure and innovative smart cloud solutions to its users. Learn more about the Azure data centers nearest to you by checking out Azure Geographies.

Azure Regions

As you can see on the Azure geographies, there are data centers all over the globe in different regions and countries. Azure regions play a vital role in cloud computing for adaptability because each country or region have its unique and different restrictions, policies, compliance, and rules. Figure 2-8 shows an example of an Azure Region with three Availability Zones.

Of all the leading cloud providers globally, Azure has the most global geographical regions. Creating your resources in the nearest Azure geographical region is an advantage. With the global and international market we have these days, being able to choose the location and geography of your resources give flexibility and reliability. Choosing the right Azure region is also important when it comes to regulatory compliance and data policies like for example the General Data Protection Regulation (GDPR) in Europe.

Before adding resources in Azure, your organization can check first if there are any legal standards or compliance that you need to comply with in your geographical area. Microsoft Azure has a list of compliance offerings for different sectors globally that you can review and learn more about.

Figure 2-8. Example of an Azure Region with three Availability Zones

Azure Region Pairs

Typically, Availability Zones are created by using one or multiple data centers. There is a minimum of three zones within a region. However, what if two of the zones go down because of a huge outage? Two data centers in an outage in the same Azure region would be risky to keep the resource in operation. This is the reason why users also have the option to use Azure Region Pairs.

Azure Region Pair, as shown in Figure 2-9, means pairing a region with another region within the same geography. By pairing regions within the same geography, users can replicate their resources like servers, virtual machines, databases and storage in another location in case of big power outages, nature calamities, etc.

If a region in a pair goes down, the services would failover automatically to the other region in its region pair.

Figure 2-9. Example of Azure Region Pair

An Azure region, as shown in Figure 2-9 is often pared with another region in the same geography. For example if the region is in Europe, then it is expected that its region pair is also located within Europe or at least 300 miles away. When it is possible, Azure prefers this minimum distance of between physical data centers in a regional pair in case of natural disasters, power outages, etc. The Azure regional pair should be within the same region to for reliability in case of unexpected interruptions.

Best Practices in Azure Region Pair

Check out Azure’s Business continuity and disaster recovery (BCDR) for Azure Paired Regions on https://docs.microsoft.com/en-us/azure/best-practices-availability-paired-regions

Azure Availability Zones

If your organization wants to make sure that your applications and cloud resources in Azure are redundant in case of failure or problems, Azure Availability Zones are your right solution. We can say that these zones are unique physical location or data center within an Azure region that offers 99.99% SLA for virtual machines uptime. An Azure Availability Zone is made up of a single or multiple datacenters with security, independent power, cooling, networking, etc.

It is important to take note that Availability Zones are not available in all regions and Azure services. SQL Managed Identity, Azure App Services and Azure Virtual Machines (Azure VMs) are supported.

Azure services supported by availablitity zones are meant to deliver resiliency, low latency, flexibility and scalability. Learn more about the services that are supported by Azure Availability Zones by region by checking out Azure Availability Zones_.

They are physically isolated data centers that are located within the same Azure region, as shown on Figure 2-10.

Figure 2-10. An example of Availability set of Azure Virtual Machines in an Azure Region.

An availability zone is usually composed of one or multiple data centers with independent facilities for power, networking, cooling, and support. The purpose of this isolation is to make sure that in case one of the zones stops working properly, the other zone continues to run in operation. These availability zones are connected and equipped with secure high-speed networks, which is important in running mission-critical resources for your computing, networking, storage, and data.

Availability Zones has a similarity to Availability Sets in Virtual Machines. In contrary, there is an obvious difference between them. Availability zones are used to protect resources from complete system failures in an Azure data center while Availability sets are used to protect applications from hardware failures within an Azure data center.

Note

If your organization has the requirement, you can replicate or transfer your data or resources into another availability zone for a cost in Azure. For example, Azure VMs can be moved to an availability zone in a different region.

Cost Management in Microsoft Azure

Aside from organizing cloud resources using Azure Management, Resource Groups, Azure tags, and making sure that your resources are in the right Azure region, you can also manage the financial aspect of your organization’s cloud consumption.

Azure’s cost management tools help the organization monitor their expenses on their cloud resources consumed. These can be done by setting budget alerts and notifications to the appropriate billing or accounting team and also estimating possible monthly or yearly costs beforehand, which aids in cost planning and budgeting.

Azure Cost Management + Billing

An administrative section in Microsoft Azure where billing and management of costs can be controlled and monitored.

Total Cost of Ownership (TCO) Calculator

A great tool that helps organization estimate the cost savings by |https://azure.microsoft.com/en-us/pricing/tco/calculator/[migrating workloads to Azure estimations].

Azure Pricing Calculator

A dedicated website where you can configure and estimate the costs for Azure products and features based on your use case scenarios in our projects. You can save, export and share these cost estimations, shown in Figure 2-10

Figure 2-11. Estimate costs for Azure services or resources using Azure Pricing Calculator

Learn more about managing costs in Azure

Get an overview and learn some of the best practices in Azure billing and cost management by visiting the Microsoft Documentation https://docs.microsoft.com/en-us/azure/cost-management-billing/

Basic Concepts of User Identities, Roles, Active Directories in Azure

In addition to structuring cloud resources using Management Groups, Subscriptions and Resource Groups, it is also important to control user identities and access to these resources.

Azure Active Directory (Azure AD) as shown on Figure 2-12 is the identity and user access management service of Microsoft. The users of Azure AD can manage their users identities, their roles, logins and access to internal resources as well as permissions to external services like Azure Portal, Office 365, and other applications.

Figure 2-12. A basic example overview of how Azure Active Directory

Azure Role-Based Access Control (Azure RBAC)

It’s important for any organization to be able to manage the user access for cloud resources and resource groups in Microsoft Azure. The Azure Role-based Access Control(RBAC) helps in user access management of resources in Azure. RBAC helps in controlling what users can do.

Azure RBAC is a system for authorization and access management of resources within the Azure platform. You can filter what a certain group of users can do and cannot do depending on the type of role they have in the organization.

For example, allow database administrators to only manage Azure resources related to databases within a resource group. The access can be set up for developers or software engineers who works with application development in the platform.

To learn more about Azure RBAC, watch this video Azure RBAC : The deep dark secrets of role based access control.

Azure Roles

In order to control access to resources in Azure, you need to set up and enforce user permissions by assigning Azure Roles. A role assignment has elements such as Security Principal, Role Definition, and Scope, as shown on Figure 2-11.

Security Principal

A security principal is an object that represents a security identity that can be authenticated and authorized to access resources. Security principals are used in Azure roles to grant or deny resource permissions. When you assign roles to a security principal, you’re granting or denying permissions to access specific resources in Azure. For example, you could set the “Contributor” role to a user principal, allowing users to create and manage resources in a specific Azure subscription. They are crucial to managing access to resources in Azure. Understanding their role in Azure roles is critical to ensuring the security and compliance of your Azure environment.

Role Definition

A role definition sets permissions for Azure users or service principals to utilize Azure resources. Role definitions define permissions for security principals to access resources in Azure. Each role definition has a set of access controls, or actions, which helps in determining which Azure resource activities can be done. A role definition may allow reading, writing, and deleting Azure subscription resources. Azure role definitions offer security principals permissions. A security principal’s role permissions apply to the assigned resources. The built-in role definitions, such as “Contributor” and “Reader,” address common resource access scenarios. Customize these built-in role descriptions or create new ones to match company needs. You also have the option to customize role definition, which lets you fine-tune resource access to meet your organization’s needs. Azure role definitions allow flexible and granular resource access management.

Scope

The scope of a role determines the level at which the role assignment applies. The scope defines the set of resources the role assignment applies to and can be set at various levels in the Azure resource hierarchy, including the management group, subscription, resource group, and individual resource levels. You can minimize the risk of unauthorized and unknown access and unintended consequences by specifying the scope of a job assignment. Managing Azure resource access and ensuring security and compliance requires an understanding role assignment scope.

As described in the previous section, the four levels of scope are management group, subscription, resource group, or resource. Also described in details on previous section Architecture and Resource Management Concepts in Microsoft Azure are scopes structured from top to bottom and follows the inheritance of parent-child relationship.

Figure 2-13. Example of the important elements of Azure Role Based Access (Azure RBAC)

Learn by Doing (Try it out!)

Now that you have learned about the cloud services in Azure, I recommend to try this section by practicing the hands-on exercises that will help give you a walk through and understanding.

Microsoft Azure Portal Hands-On

If you already have an Microsoft Azure Subscription, visit https://portal.azure.com

If you don’t have a subscription yet, register for free Azure account on link https://azure.microsoft.com/en-us/free/

Explore the Azure Portal’s Home Page, as shown in Figure 2-14 is where you will see the left side bar with the default list of Azure Services and the main page has the dashboard and search bar. On the upper-right of the portal, you will be able to access the Cloud Shell, Directories + Subscriptions, Notifications, Settings, Help and Manage your Azure Account.

Figure 2-14. Example of how the page of the Microsoft Azure Portal look like when you signed up for an Azure account and have logged in

Click on All Services on the left panel and explore the cloud services in different categories as shown in Figure 2-15.

Figure 2-15. All services in Microsoft Azure Portal in different categories

Find Free Offerings in Microsoft Azure in the same page or visit https://portal.azure.com/#blade/Microsoft_Azure_Billing/FreeServicesBlade Explore the free Azure services available for you for 12 months with your Azure free account.

Learn By Doing (Try it out!)

There are a few hands-on exercises included on the supplementary repository for this chapter.

Summary

In this chapter, you have learned the fundamentals of public cloud provider, Microsoft Azure. You learned about the different Azure services in different categories e.g., Compute, Storage, Networking, Databases, Developer Tools, Integration, Analytics and more. Resources in Azure can be organized by and structured in to different scopes Azure Management Groups, Subscriptions and Resource Group. Azure has helpful tools to manage costs and also protection your data and resources in case of outages using availability zones and Azure Region Pairs. Azure Active Directory (Azure AD) allows you to manage users and identifies to control operations of your Azure resources.

Azure Role-based Access Control (Azure RBAC) is the user access control and identity management in Azure and that a role assignment is composed of three important elements - security principal, role definition, and scope. We also learned that Azure Resource Manager(ARM) templates and Terraform are used for automation of deployments to Azure and for implementing Infrastructure as a Code (IaC).

On final note, the core Azure services mentioned on this chapter will be described in detail in rest of chapters of this book.

Check Your Knowledge

Can you have two Azure region pairs in different geography? (True or False)

Azure Marketplace is where Azure users can purchase and try-out applications and services from other service providers and Microsoft partner companies. (True or False)

What are the top categories of core services in Azure?

Azure Virtual Machine Scale Sets is the compute service you need if you want to develop event-driven applications in serverless environment. (True or False)

What is the difference between Azure Region vs. Availability Set?

Answers

To find the answers to the Check Your Knowledge, turn to back of the book in the Appendix

Recommended Resources

Azure Global Infrastructure

What is Azure?

Microsoft Azure Fundamentals Learning Path

Microsoft Documentation - How to Create an Azure Account

Microsoft Learn - Control and organize Azure resources with Azure Resource Manager

Directory of all Microsoft Azure Services

Azure Charts

Microsoft Documentation: Check access for a user to Azure resources

Microsoft Learning Path - Build a cloud governance strategy on Azure

Azure Blob Storage Documentation

Microsoft Learning Path - Describe Azure Core Services

Martin Kleppmann, 2017. “Designing Data Intensive Applications”, O’Reilly Media, Inc. ISBN: 9781449373320

What is Azure role-based access control (RBAC)? | One Dev Question: Arturo Lucatero

Azure Real World Hands-on Training for Beginners by Ryan O’Connell

¹ As of time of writing

Part II. Cloud Computing Services, Networking, Storage and Databases in Microsoft Azure

This second part of the book and its chapters focus on giving you an in-depth look at the different technologies in Microsoft Azure for categories Compute, Networking, Databases and Storage. These categories relate to each other and they are widely used in workloads, infrastructure and applications in the cloud.

Chapter 3 explores some of the Microsoft Azure compute services such as Azure Virtual Machines, container services like Azure Container Instances, Azure Container Registry, Azure Container Apps, Azure App Services, for web and mobile applications, serverless cloud solutions with Azure Functions, Azure Static Web App, Quatum Computing, etc.

Chapter 4 covers cloud networking and services in Microsoft Azure including Azure VNet, DNS, Azure Firewall, Azure Front Door, ExpressRoute, Virtual Network, VPN Gateway, Application Gateway, Load Balancer, Internet Analyzer, etc. You will also get an introductory overview of Azure Space and Azure Orbital which is Azure’s Ground Station as a Service.

Chapter 5 provides a technical overview of the different cloud storage and databases (both SQL and NoSQL) in Microsoft Azure. You will learn about cloud storage concepts, services, create databases and find useful best practices for Azure SQL Databases, Azure Cosmos DB, Database for MySql, Azure SQL Servers, Redis Cache in Azure, Azure Storage, Data Share and Manage Disks.

By the end of reading Section 2, you already have a good understanding of the different networking, cloud storage, and databases in Microsoft Azure which will help you decide which suits best in the specific use case of your cloud development projects.

Chapter 3. Microsoft Azure Cloud Compute Services

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 3rd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at jleonard@oreilly.com.

Compute services, networking, and storage are the essentials of any public cloud. Azure Compute helps customers build and host application workloads of various scales and natures. It works with your current development stack or helps migrate it to containers or serverless and adopts the best development practices.

Alexey Polkovnikov, Sr. Cloud Solution Architect at Microsoft & Founder of Azure Charts

Previously, we learned about Microsoft Azure as a cloud provider. We also had a good overview of Azure’s different core services, which gave us a better understanding of each cloud service.

In this chapter, we dig deeper into one core services category in Microsoft’s cloud platform, Azure Compute, that helps you and your organization build cloud solutions using compute services, which be in existing development projects regardless of whether your applications are for migration, containerization, modernization, and more.

Azure Compute for Developing Fully-Managed Systems

Azure provides computing services on-demand globally and with high availability. Azure Compute is the category name used for the cloud-based computing services in Azure. It is one of the service categories in Azure that offers on-demand computing resources such as high-capacity virtual disks, memory, networking, processors, and robust virtual operating systems. They are available on a Pay-As-You-Go and other pricing models that can be set up and created quickly. The ability to provision compute resources with flexibility of pricing and scalability can be challenging if you are provisioning compute resources on-premises and traditional IT infrastructure.

You find the different compute services in Azure Portal by navigating to All Services and then selecting the category Compute as shown in Figure 3-1

Figure 3-1. Overview of some of Azure Compute Services in Azure

Note

Azure Compute services are used for running and hosting the workloads for our applications. Some of the familiar Azure compute resources are Azure Virtual Machines (VMs), Azure VM Scale Sets, Azure App Services, Azure Container Apps, Azure Functions, Azure Kubernetes (AKS), and other computing services., as shown in Figure 3-2. These compute resources are categorized in different deployment models,s such as Infrastructure as a Service(IaaS), Platform as a Service (PaaS), and used in combination with Serverless or separately depending on the type of applications and business solutions that need to be solved.

Figure 3-2. Common Azure Compute services in Azure

These computing technologies will be useful in hosting your applications in the cloud while gaining the benefits of having resources and workloads fully managed, better security, and opportunity for rapid development and innovation that are still on-premises can make use of Azure Compute services to improve their systems and save time and money from the traditional IT infrastructure.

Azure Virtual Machines and Virtual Machine Scale Sets

Aside from the commonly used compute service, there is another way to deploy applications and databases to Azure using Azure Virtual Machines and Azure VM Scale Sets that provide opportunities for hybrid deployment of both on-premises and cloud. This enables you to deploy your applications and databases to Azure in a more cost-effective and scalable way than traditional IT infrastructure.

Two of the most popular services in the Azure Compute category are the Azure Virtual Machines (Azure VMs) and Azure Virtual Machine Scale Sets (Azure VM Scale Sets). These services enable us to quickly create virtual machines on the cloud on demand anytime and anywhere, especially for those who prefer implementing IaaS solutions for their business needs.

For example, Azure VM Scale Sets allows you to create thousands of Azure VMs on the cloud in a few minutes with auto-scaling and load-balancing features that will help you meet the demands of your workloads. Azure VM Scale Sets can also be used as Azure DevOps Agents for CI/CD pipelines.

Azure VMs are compatible with operating systems on different platforms like Windows, Linux, and others. Azure VMs provides high availability, scalability, reliability, and fully-managed VM management features for its users and administrators. They are also commonly used by those seeking for Lift-and-Shift cloud migration scenarios.

The Lift-and-Shift cloud migration is known to have critical benefits such as minimal code-refactoring, re-architecture, and faster migration. This cloud migration strategy is ideal for those organizations who want to move their existing on-premise IT infrastructure to the cloud without making significant changes to the existing architecture, source code, and processes. By utilizing IaaS solutions and even container solutions for cloud migration, organizations can save time, money, and resources by re- re-structuring and modernizing the applications.

Cloud migration options and services in Azure will be discussed in greater detail in Chapter 13 Microsoft Azure Cloud Management and Governance.

As you may recall from Chapter 1, virtualization co-relates to the evolution of cloud computing, and it is what drives the IaaS solutions we have today.¹.

Azure Virtual Machines

As mentioned, Azure VMs are considered an Infrastructure as a Service (IaaS) cloud service model, which means that you have complete control of the Infrastructure plus the advantage of scalable computing resources. Having complete control of the Infrastructure also means that we are responsible for maintaining it.

Azure VM may be ideal if your organization wants to have full capabilities in installing and configuring your Infrastructure without spending money buying physical hardware or expanding data centers. Additionally, Azure has options to allow Azure VM users to manage security, monitoring, updates, and patches to the operating systems.

Through virtualization in the cloud, you can deploy and host your applications on your desired OS on VMs in different servers and with shared storage. Figure 3-3 shows a simple example diagram of running an Azure VM on Azure based on Microsoft Documentation.

Figure 3-3. An example overview of Azure Virtual Machines

Development and Applications

Through its cloud-based services, Azure VMs provide you with the flexibility of virtualization, and it also gives you control over how you want to host your applications. A wide range of computing solutions exists for development, testing, running applications, and extending your data center. It gives some users the option of using custom and open-source software systems the way you need to configure them. This gives the flexibility and benefit of faster application deployments in seconds instead of weeks.

Modern Cloud App Development with Azure VMs

Azure Lab Services and Azure DevTest Labs are useful if you want to set up different development, test, and lab environments for your team or for learning purposes.

Making use of Azure VMs as part of an IaaS solution is useful, especially if you want to take control of your on-premises resources or Infrastructure. However, there are some important and practical things to consider before provisioning Azure VMs for your organization on the cloud, as described in Table 3-1.

Table 3-1. Things to Consider Before Creating Azure VMs

Consideration
Description

Basics of Networking

Azure uses virtual networks (VNets) to secure connections for Azure VMs and other cloud services in the platform. Any Azure VNet Resouces that belong to the same private network can be accessed

Name of the VM

Choosing the appropriate and consistent name of your Azure VM is part of the provisioning process. The VM name is configured as part of the operating system and is also used to define and manage it as a resource

VM Size

Options to choose VM sizes are available so that you can choose the right combination of memory, compute, and storage for type workload needed

Knowledge of VM Pricing Models

Being aware of the pricing models for Azure VMs is important to prevent unwanted costs. Azure VM pricing models come in five different types. Each type of option will vary not only in price but also in performance and availability

VM Storage

Azure VMs usually will have at least two virtual hard disks (VHDs) which is used for the storage of the operating system(OS) and the temporary storage. Determining the appropriate VM storage prior to provisioning is good to know so that the VMs you create are provisioned to handle the workloads you need. Check out Microsoft’s recommendation for Virtual Machine Sizing Guidelines.

Choosing the right Operating System

There is a variety of OS images that you can install into the Azure VM, including several versions of Windows and different types of OS in Linux. Choosing the appropriated OS prior to VM provisioning is vital. Usually the choice of OS will affect the cost of hourly compute pricing as Azure combines the cost of the OS license into its price. Using Azure Cloud Shell, you can easily see the list of latest Azure VM images and OS versions using the command az vm image list and other required parameters.

For more details and information about Azure VM provisioning, visit Microsoft’s checklist for creating an Azure Virtual Machine and evaluate the different pricing models for Linux VMs and Windows VMs

Azure Virtual Machine Scale Sets

Azure Virtual Machine Scale Sets (Azure VM Scale Sets) is a compute service that allows you to deploy, run and manage a scalable set of multiple virtual machines (VMs). With all VMs configured the same, Azure VM Scale Sets are designed to support autoscaling capabilities. There is no requirement for pre-provisioning of VMs which makes it easier to build services at a large scale. This can be useful in developing solutions with big data, big computing, and even workloads that are containerized.

When the demand for workloads increases, there is an option to add extra instances of virtual machines. When the need for workload decreases, VM instances can be deleted or scaled down. The scaling options and processes for Azure VM Scale Sets can either be done automatically or manually, depending on your workload needs. It is also possible to dynamically combine autoscaling and manual scaling if necessary.

You can set autoscaling options on Azure VM Scale Sets, as shown in Figure 3-4.

Figure 3-4. A simple illustration of vertical and horizontal scaling

Scaling Options for Azure VM Scale Sets

Azure VM Scale Sets are meant to help us save some money from the risk of having high costs in having multiple virtual machines in the cloud. Therefore, new instances of virtual machines are recommended to be created or provisioned when necessary.

In Azure, there is an autoscaling option to scale up and down the number of compute resources that are being allocated based on its demand at a certain time. You can scale your virtual machines using horizontal scaling or vertical scaling.

A good example of typical horizontal scaling is when you add or remove some VMs from a scale set. On the other hand, with vertical scaling, you are adding additional resources to your VMs, like upgrading the memory, disk memory, and CPU power.

Azure Hybrid Benefit and Azure Spot Virtual Machines

As your workloads and VMs increases in the IaaS model, using Azure VMs can be expensive. Azure has optional services like Azure Hybrid Benefit and Azure Spot VMs. Azure Hybrid Benefit is a licensed benefit that helps organizations save money by reducing the costs of running workloads in the cloud. The benefit gives the option for organizations to use their own on-premises Windows Server and SQL Server licenses that are Software Assurance-enabled on Azure.

The Azure Hybrid Benefit Savings Calculator is also a useful tool for analyzing the financial benefits you will gain. Learn more about how this licensing benefits and save money on Azure VMs by availing and making use of your organization’s on-premises licenses for Windows Server and SQL Server.

In addition to this, Azure Spot VMs can help saving costs through unused capacity.

Finally, both Azure VMs and Azure VM Scale Sets are two great options for hosting databases, storage, and applications for your organization. They are both ideal, especially if your organization is still in the process of getting started in your cloud migration journey and prefers to have some of your resources, data, and workloads on-premises.

Azure App Service

Azure App Service is considered a Platform as a Service (PaaS) and is an HTTP- based service that enables us to host web applications on the cloud. If you do not have the need to have full control of your Infrastructure, like the demand for using Azure VMs, then App Service is one of the most commonly used fully-managed PaaS services for web applications development.

It allows developers to easily create, deploy, and manage cloud-based web apps and APIs. With App Service, developers can concentrate on writing application code rather than fretting about the infrastructure that underlies the application.

App Service has the following applications and features:

Built-in Integration Capabilities

App Service offers built-in integration with central source control systems such as GitHub and Azure DevOps to facilitate deployment. Developers have various application deployment options, including code deployment, Docker containers, and Azure Functions integration. The feature of flexible integration with other Azure Services such as Azure SQL Database, Azure Blob Storage, and Azure Functions. It helps developers create a comprehensive solution that is a scalable, dependable, and secure solution.

Support for Multiple Languages and Frameworks

This service supports multiple programming languages and frameworks. NET, Node.js, Java, PHP, and Python. The support for various programming languages and frameworks allows developers to select the language and framework best suits their project’s needs.

Autoscaling

App Service can autonomously scale up or down application instances based on traffic load. The autoscaling features help ensure that the application’s efficacy is maximized while costs are minimized.

High Availability and Fault Tolerance

It also offers high availability and tolerance capabilities for failures, including geo-redundancy, automatic failover, and backup and restore capabilities. This guarantees that the app is always available to users and can recover rapidly from failures.

Hybrid Connections

Azure App Service provides a feature called Hybrid Connections, which allows developers to securely connect their App Service applications to on-premises resources over the Internet without requiring complex configuration or infrastructure changes.The Hybrid Connections feature creates a secure outbound connection from the App Service to a relay agent hosted in the Azure Relay service. The relay agent acts as a bridge between the App Service and the on-premises resource, allowing the App Service to communicate with it securely. To use this hybrid integration feature, the developer must set up a relay agent in Azure and configure it with the appropriate settings. Once it is set up, it can connect the App Service to an on-premises resource like a database, web service, or TCP endpoint. This is accomplished by configuring the connection string to help use the Hybrid Connection endpoint. This feature of Azure App Service is handy for enterprises with legacy systems or on-premises resources that cannot be easily moved to the cloud. It allows them to leverage the benefits of Azure App Service while still being able to access their existing systems and data. Additionally, it eliminates the need for complex VPN configurations and firewall rules, making it easy to use and manage.

Overall, App Service is a powerful PaaS solution that offers a variety of features and capabilities for developing and deploying web-based applications and APIs on Azure.

Figure 3-5, shows a simple example architecture from Microsoft Documentation website with implementation of an App Service in an App Service Plan. . The Web app has integrations with other services like Azure Active Directory for authentication, Azure Front Door, Queue for a message, Azure Functions for serverless back-end logic to database storage to Azure SQL and Cosmos DB databases using Redis Cache.

Figure 3-5. A basic example of a web application implementing Azure App Service in different environments Dev, Test, and Production using Deployment Slots. The application has its Azure SQL Databases on SQL Server and integrates with Azure Monitor for logging the metrics and diagnostics for the web application.

Modern application development is more efficient using App Service because it has features good for web applications for autoscaling, automation, security, workload balancing, monitoring, and support for continuous delivery and continuous integration (CI/CD) with Azure DevOps, Docker Hub, Github, etc. The list of key benefits of using Azure App Service is listed and described inTable 3-2.

The Azure App Service plan determines the type of pricing model and how much you pay for the Azure compute resources consumed monthly.

Table 3-2. Benefits of using Azure App Service

Benefits
Description

Multiple languages and frameworks

App Service has first-class support for different frameworks and programming languages ASP.NET, ASP.NET Core, Java, Ruby, Node.js, PHP, or Python. You can also run PowerShell and other scripts or executables as background services

Managed environments

App Service automatically patches and maintains the OS and language frameworks which enable developers to spend more time writing code and worry-less about setting up the platform

Containerization

Containerized or dockerized your application and hosted it on a custom container (Linux or Windows) in App Service. Using Docker Compose, you can deploy and run multi-container apps

Optimized DevOps

Set up continuous integration and deployment with Azure DevOps, GitHub, BitBucket, Docker Hub, or Azure Container Registry. Promote updates through test and staging environments. Manage your apps in App Service by using Azure PowerShell or the cross-platform command-line interface (CLI)

Global Scale and high availability

Host your apps anywhere in Microsoft’s global data center infrastructure with autoscaling or manual scaling capabilities, and high availability

Integration with On-Premises and SaaS Platforms

A wide variety of connectors are available for enterprise systems, SaaS services, and external services. Data from on-premises systems can also be accessed using features like Hybrid Connections and Azure Virtual Networks.

Security and compliance

You can secure your App Service by adding Azure Active Directory for user authentication or Single Sign On (SSO) with external services like Google, Facebook, Twitter, or Microsoft account. You can set-up Managed Service Identities and restrict some IP addresses for security

Application templates

If you need to use app templates, there are a list of them in the Azure Marketplace, such as WordPress, Joomla, and Drupal

Integration with Visual Studio and Visual Studio Code

Development team can code, deploy and debug using the great IDEs like Visual Studio and Visual Studio Code

API and mobile features

App Service provides turn-key CORS support for RESTful API scenarios, and simplifies mobile app scenarios by enabling authentication, offline data sync, push notifications, and more

Serverless logic

Web apps can be integrated well with serverless code like used in Azure Functions. This means that you can run a serverless code, scripts and call API endpoints on-demand without having to manage Infrastructure or explicitly provisioning. Using the consumption plan pricing model, you only pay for the compute time when your Azure Functions instance runs.

So, we learned that Azure App Service allow us to develop and host our applications and APIs using our preferred the programming language as PaaS without managing infrastructure. App Service also supports single-page application (SPA), open source content management systems (CMS) like WordPress, Apps on Azure Arc, Static HTML sites, multi-container apps using Docker, etc.

Static Web Apps

Azure Static Web Apps is a type of single-page application service which allows you to develop full stack SPA web applications that complete CI/CD integration with a code repository platform like GitHub or Azure DevOps. Static Web Apps are known to be reliable, quick and fast applications. A few of the prerequisites in developing Static Web Apps are an Azure and GitHub Account since the source code needs to be setup with GitHub Actions for the automation of the deployment workflow.

Static Web App in Azure gives us the option to build modern web applications with modern frameworks and libraries we have today. For example, this Azure service can be integrated with Blazor Web Assembly applications, React, Angular, Vue, or Svelte. It is also designed to integrate well with serverless development by using Azure Functions as back-end or for APIs. If you need guidance in choosing between the traditional web applications compared to using SPAs, check out this good Microsoft documentation, Choose Between Traditional Web Apps and Single Page Apps (SPAs) about it.

Figure 3-6. Azure Static Web Apps Example Overview

Figure 3-6 shows how Azure Static Web Apps workflow happens after updating to a source code. Any code changes trigger the deployment workflow using GitHub Actions or Azure DevOps Pipelines.

Key Benefits and Uses of Azure Static Web Apps

There are many ways and technologies to create modern application in the cloud these days. If you think your application must expose a rich user interface with many features. Static Web Apps doesn’t require reloading the page as users take actions or navigate between areas of the app.

The use cases and benefits to using Static Web Apps are as follows:

Build modern web applications WebAssembly application like Blazor, and more.

Bring Your Own Functions Support for serverless development especially if you already have existing Azure Functions to implement

Seamless developer experience and CI/CD with GitHub Actions or Azure DevOps

Globally distributed static content closer to location of the users of your web applications

Free and auto-renewable SSL certificates for both Free and Standard pricing models

Custom domains support with TLS

Seamless security with support of reverse-proxy when calling APIs, which requires no CORS configuration ²

Authentication with common identity providers like with Azure Active Directory, GitHub, Twitter, etc.

Pricing Plans for Azure Static Web Apps

Free and Standard plans are the two different pricing models to choose for Static Web Apps. Choosing the appropriate plan depends on the requirements and use cases. Learn more about the list of features of each plan on Azure Static Web Apps hosting plans and be sure to reference the quotas page before determining the best fit for your needs.

Oryx is the build engine of Azure Static Web Apps. It helps in building the frontend and the API in your web application. It is a smart build engine because it automatically builds the steps and executes them for you depending on the type of framework you have implemented in the app.

Frontend Framework Configuration

Since Azure Static Web Apps dynamically allow us to build SPA using the frontend framework of our choice, the differences on the configuration exists. The build configuration values needs to be specific to the type of framework or library is used for the frontend.

Learn more about it Configure front-end frameworks and libraries with Azure Static Web Apps

In conclusion, Azure Static Web Apps are idea for modern SPAs and for hosting static web content with support for globally distribution, scalability, speed, security, integration, and ease of use for both development and deployment.

Azure Web App for Containers

Azure Web App for Containers enables you to deploy your applications as containers in the cloud with the capability to use custom docker images with support for autoscaling and easy deployment. Container registries like Docker Hub or Azure Container Registry can be used to pull the images of your application. An example architecture of Azure Web App for Containers is shown in Figure 3-7.

Figure 3-7. Example architecture of Azure Web App for Containers.

The Web App for Containers in Azure also has the feature to develop a custom container and multi-container (Preview) apps using Cloud Shell and Docker.

Serverless Compute Services

One of the powerful and promising compute services in Azure is the serverless compute services. Since the technology of Serverless computing came, developers had the opportunity to be effective by building applications without maintaining and worrying about the underlying infrastructure. Technically, the servers and infrastructure of applications in serverless compute are automatically provisioned, which means they are managed and scaled by Azure, the cloud service provider. As a developer, this helps you focus on writing the code and the application development.

Benefits of Serverless Architecture in the Cloud

In simple terms, serverless computing is a model where backend services are provided as a cloud service.

For example, an organization can make use of the serverless or backend services from Azure with the advantage of having a consumption type of pricing model, which means you pay as you use. Figure 3-8 illustrates how serverless computing can help save money from building and maintaining servers and infrastructure the traditional way.

On-premises IT infrastructure is expensive to have because it relies on our own physical hardware and data center to host our systems and web applications on our own servers.

Figure 3-8. An illustration of how serverless computing helps in cost savings in IT infrastructure

Because serverless providers like Azure provides autoscaling capabilities, they can help reduce the cost of maintaining physical servers or data centers.

Azure Functions

Azure Functions is a powerful compute service for serverless computing. Serverless functions in Azure are ideal when you’re concerned only about the code running your service and not the underlying platform or infrastructure. For example, you can use Azure Functions for the backend logic of your APIs or integration with other services or APIs. This compute service is event-driven, which means it can be used if you need to do tasks in response to an event. For example an HTTP triggered message arriving in an Azure Service Bus Queue that passes the queue message to another enterprise application or service.

Benefits of Azure Functions

Write less code by focusing on writing the logic you need to write

Focus on development and delivery

Worry less on maintaining the servers and infrastructure

Save on costs by paying only when functions are running

Flexibility of using different languages and frameworks supported

Autoscaling

Wide variety of integration options with other services

Components of Azure Functions

Azure Functions is designed for event-driven applications. It has two components as the following:

Triggers

An Azure function will not run without the help of triggers. A trigger based on any event defines how an Azure function is invoked. It is important to take note that an Azure function must have only one trigger and the triggers in a function can have corresponding data usually serves as the payload.

Bindings

To declaratively connect another resource to the function; bindings are optional and may be connected as input, output, or both.

Both the triggers and bindings let you avoid hard-coding access to other services. Your function receives data (content of a queue message) as parameters in your function.

When you create and configure your Azure Function, it is worth to know that all triggers and bindings have specific direction_property in the functions configuration file function.json. Triggers always has the in direction and the bindings use in and out depending if it was an input or an output. There is another direction called inout which is supported in some bindings. Learn more about Azure Functions triggers and bindings.

Create and Build Azure Functions

Azure Functions can be developed in different programming languages like C#, Java, PowerShell, Javascript, Python, Typescript, etc. Try building your first Azure Functions via Microsoft’s Getting Started with Azure Functions.

Azure Durable Functions

Durable Functions is an Azure Functions exten users to develop stateful functions within a serverless computing environment. These functions can be executed without the need for a server. It has the same standard functionality as Azure Functions, but its primary essential purpose and use ar for constructing serverless stateful workflows and orchestration. Creating and developing your logical workflows with the help of orchestrator functions and stateful entitiesating complicated challenges.

Developing stateful workflows with Durable Functions is beneficial in certain use case scenarios like automating business processes like approval workflow that requires a human response or interaction. For example, when you have a scenario that requires you to wait for an external event. Durable Functions helps in keeping workflows stateful by managing state, checkpoints, and restarts the workflow for you. Because of its benefits, developers gain productivity and can focus on writing the code, implementing the business logic and delivering the business requirements.

Durable Functions as of date of writing supports the following languages like C#, JavaScript, Python, F#, and PowerShell.

Note

The Durable Task Framework (DTFx) is the library behind Azure Durable Functions. DTFx allows its users to write long running (stateful) and persistent workflows (orchestrations) in programming language C#.

Learn more about this framework on GitHub and watch this interesting video about it Building workflows with the Durable Task Framework on On.NET.

Key Features and Benefits of Azure Durable Functions

Take control of complex use case workflows

Solve event-driven scenarios in your applications

Develop stateful orchestration workflows of your functions or tasks in serverless environments

Worry-less on maintaining infrastructure

Pay only for instances when your function apps are running

Integration flexibility with other Azure services and APIs

Components of Azure Durable Functions

Azure Durable Functions have components or function types that play important roles in the serverless stateful workflow and orchestration of tasks. A typical durable function app in Microsoft Azure is usually composed of different types of Azure Functions. Each of them can be stateful or stateless depending on the type of you are using and what it should be doing.

Durable functions have four types: the client functions, orchestrator functions, activity functions, and entity functions, as illustrated in Figure 3-9.

Figure 3-9. The known function types of Azure Durable Functions

Let’s take a look at the details of each of these durable function types to better understand how it all works altogether in authoring and developing stateful workflows or orchestration.

Client Functions

Client functions perform a big role in starting the orchestration of durable functions workflow. It needs to have an orchestration client binding to trigger the instance of the orchestrator functions.

Any Azure function that is non-orchestrator can be considered as a client function. For example, you can trigger and start the orchestration of your workflow using HTTP trigger, Azure Cosmos DB trigger, Azure Service bus trigger, and other triggers supported by Azure Functions framework. The difference is that a normal function has from a client function is that it uses a DurableClient output binding as shown in code Example 3-1 which is an example from one of my serverless development and demo projects I published on my GitHub repository for one of the application patterns, Function Chaining which will be discussed in details further in this chapter.

Example 3-1. An example of a client function that use the Azure BLOB trigger to start the orchestration of the durable workflow by calling the orchestrator function called StorageOrchestrator

[FunctionName("BlobTriggerStart")]
    public static async Task BlobTriggerClientFunction([BlobTrigger("photoscontainer/{name}",
         Connection ="StorageConnectionString")]  CloudBlockBlob myBlob, string name,
         ILogger log, [DurableClient] IDurableOrchestrationClient starter)
    {
        try
         {
            log.LogInformation($"Started orchestration trigged by BLOB trigger.
                                     A blob item with name = '{name}'");

            // Function input comes from the request content.
            if (myBlob != null)
            {
                var blobItem = new CloudBlobItem
                {
                    Name = myBlob.Name,
                    BlobUrl = myBlob.Uri.AbsoluteUri.ToString(),
                    Metadata = (Dictionary<string, string>)myBlob.Metadata,
                    FileType = myBlob.BlobType.ToString(),
                    Size = myBlob.Name.Length.ToString(),
                    ETag = myBlob.Properties.ETag.ToString()
                };

                var instanceId = await starter.StartNewAsync("StorageOrchestrator", blobItem);
                log.LogInformation($"Started orchestration ID = '{instanceId}'.");
            }
            else
            {
                log.LogError($"The blob was trigged but myCloudBlob was empty");
            }
        }
        catch (Exception ex)
        {
            //TODO Error handling
            log.LogError("Something went wrong. Error : " + ex.InnerException);
            throw;
        }
    }

Orchestrator Functions

Orchestrator functions provide an overview of the manner in the actions are carried out as well as the sequence in which actions are carried out. As seen in the Durable Functions application patterns, the orchestration is described by the Orchestrator functions, which are written in code (C# or JavaScript). An orchestration may contain a wide variety of operations, such as activity functions, sub-orchestrations, HTTP requests, timers, and waiting for external events. The functions of the orchestrator may communicate with the functions of the entity.

Important note to know about Orchestrator Functions

The code used to write orchestrator functions resembles other types of code; however, there are specific guidelines for how the code should be written. In particular, the code for orchestrator functions needs to be deterministic. If you do not adhere to these determinism requirements, it is possible that the orchestrator functions will not run successfully. There are code constraints developers need to consider in terms of containing in-depth information on these prerequisites as well as instructions on how to get past them.

Learn more about deterministic APIs.

When the orchestrator function gets triggered by any type of client function using the the durable orchestration trigger, it starts the orchestration and runs the workflow that you authored and designed in your code.

Code Example 3-2 shows a simple example of the orchestrator function with a durable stateful workflow by calling the activity functions that does the work in greeting hello to the passed cities as an input.

Example 3-2. An example of an orchestrator function called StorageOrchestrator that uses the Function Chaining pattern. The orchestrator function checks if the BLOB data passed from the client function BlobTriggerStart is not empty or of null value before it calls the other activity functions SendMessageToServiceBusQueue, SendSmsCallviaTwilio, and SendEmailNotification.

[FunctionName("StorageOrchestrator")]
    public static async Task<string> RunOrchestrator([OrchestrationTrigger]
                            IDurableOrchestrationContext context, ILogger log)
    {
        try
        {
            var uploadedBlob = context.GetInput<CloudBlobItem>();
            bool isEmailSentToAdmin;

            //Chain #1 Send Message with BLOB details to Service Bus Queue
            var queueMessage = await context.CallActivityAsync<string>(
                                        "SendMessageToServiceBusQueue", uploadedBlob);

            if(queueMessage != null)
            {
                //Chain #2 Send SMS and call via TwilioAPI
                var isSmsCalledUser = await context.CallActivityAsync<bool>(
                                            "SendSmsCallviaTwilio", serviceBusMessage);

                //Chain #3 send email using Sendgrid API
                if (isSmsCalledUser)
                {
                    isEmailSentToAdmin = await context.CallActivityAsync<bool>(
                                                     "SendEmailNotice", uploadedBlob);
                }

                log.LogInformation($"A new blob named {uploadedBlob.Name} was queued" +
                                    $"and added to service bus queue. \n" +
                                    $" SMS sent = {isSmsSentAndCalledUser} to user. \n" +
                                    $" Access via URL: {uploadedBlob.BlobUrl}" +
                                    $" and email sent.");
            }

            return $"Orchestration done with Id: {context.InstanceId}";
        }
        catch (Exception ex)
        {
            //TODO Handle possible errors and do a retry if needed or retry a function
            log.LogError($"Something went wrong " + ex.Message);
            throw;
        }
    }

Example 3-2 uses function chaining with a few tasks to do; however, you can actually combine the different patterns to create a more complex orchestration. Application patterns will be discussed after the components and types of Azure Durable Functions are introduced.

Additionally, in an orchestration, you also have the option to write sub-orchestrations within your orchestrations. To learn more about the important things you need to follow and know, you may read more about it on Microsoft’s Documentation for Azure Functions

Activity Functions

Activity functions are the work to be executed in a durable orchestration. They are the tasks that are orchestrated in the series of event-driven workflow. For example, you might create an orchestrator function to process an order. The tasks involve checking the inventory, sending an email to the customer, and creating a shipment process, and other event-driven related tasks. These functions can be executed in parallel, or combination, depending on the type of workflow orchestration pattern you are using for your use case.

These types of Durable Functions are short-lived functions that perform specific tasks in a workflow. They are called from the orchestrator function, and their input and output are managed by it. Activity functions allow you to break down complex workflows into smaller, reusable tasks and to handle errors and retries more efficiently.into smaller, reusable tasks and to handle errors and retries more efficiently

Another simple template example from Microsoft documentation is shown in Example 3-3. This is an example of the activity functions that receives the parameter of the string of cityName. The HelloGreeter_Activity does its job to output and greeting hello to the city once it is triggered using the ActivityTrigger.

Example 3-3. An example of the first task or activity being called by the orchestrator StorageOrchestrator. The activity function SendMessageToServiceBusQueue has the logic to send BLOB data uploadedcloudBlob and compose a string text message to be sent and saved in the Azure Service Bus queue. Once that queue message is saved in the queue storage in Azure, it will return that queue message to the orchestrator function StorageOrchestrator.

[FunctionName("SendMessageToServiceBusQueue")]
    public static async Task<string> SendMessageToAzureServiceBusQueueAsync([ActivityTrigger]
            CloudBlobItem uploadedBlob, ILogger log, ExecutionContext executionContext)
    {
        log.LogInformation($"Received data {uploadedBlob.Name}, format {uploadedBlob.FileType}.");

        //Config settings for Azure Service Bus
        var azureServiceBusConfig = new ConfigurationBuilder()
            .SetBasePath(executionContext.FunctionAppDirectory)
            .AddJsonFile("local.settings.json", optional: true, reloadOnChange: true)
            .AddEnvironmentVariables()
            .Build();

        var serviceBusConnection = azureServiceBusConfig["AzureServiceBusConnectionString"];
        var serviceBusQueue = azureServiceBusConfig["ServiceBusQueueName"];
        string composedMessage = "";

        try
        {
            if (uploadedcloudBlob != null)
            {
                log.LogInformation($"Composing message to be sent to the queue");

                composedMessage = $"A blob image {uploadedBlob.Name} was added queue </br> " +
                                         $"Blob Type: {uploadedBlob.FileType} </br> " +
                                         $"Blob URL: {uploadedBlob.BlobUrl} </br> " +
                                         $"Message sent via Azure Durable Functions App";

                await using (ServiceBusClient client = new ServiceBusClient(serviceBusConnection))
                {
                    //Create sender
                    ServiceBusSender sender = client.CreateSender(serviceBusQueue);

                    //Create message
                    ServiceBusMessage message = new ServiceBusMessage(composedMessage);

                    //Send Message to ServiceBus Queue
                    await sender.SendMessageAsync(message);
                    log.LogInformation($"Sent queue message: {serviceBusQueue}");
                    return composedMessage;
                }
            }
            else
            {
                 return composedMessage;
            }
        catch (Exception ex)
        {
            log.Error($"Something went wrong sending queue message to {serviceBusQueue}");
            log.Error($"Exception {ex.InnerException}");
            throw;
        }

    }

If look through the overview of the StorageOrchestration workflow using Durable Functions, as soon as the user uploads a blob image to the storage account, a queue message is composed to be queued to an Azure Service Bus queue which will then make some tasks of sendinge e-mail, call or SMS to the user or the administrator. This is just a simple implementation and you can do more complex things with the Durable Functions.

If you to learn about the different use cases, check out the list of interesting projects on Azure Serverless Community Library.

Entities Functions

Entity Functions are a feature of Azure Durable Functions that provide a way to manage stateful entities in a distributed, scalable, and reliable manner. An entity is an instance of a stateful object that a Durable Function can create, update, and delete.

As a developer, it allows you to define a type of stateful object as a class and annotate it with an attribute. This class can be used to determine the state and behavior of an entity, including its properties, methods, and events. The state of an entity is stored in the Azure Durable Functions state storage, designed to be durable and reliable.

Entity Functions provide a way to manage the state of entities using optimistic concurrency control. When a Durable Function accesses an entity, the current version of the entity state is retrieved from the state storage. The Durable Function can then modify the entity’s state and save the new state back to the storage. Suppose another Durable Function has changed the entity state in the meantime. In that case, the save operation will fail with a concurrency exception, and the Durable Function can handle the exception and retry the operation if necessary.

These functions help manage stateful entities in a distributed system, such as users, accounts, orders, or inventory items. Entity Functions allow you to encapsulate the logic and state of an entity in a single class, making it easier to maintain and evolve. They also provide a way to ensure consistency and reliability when multiple Durable Functions access the same entity simultaneously.

Let us review what are the uses and benefits of using Entity Functions types in Durable Functions in Azure:

Scalability: It can handle many concurrent requests and scale dynamically to meet demand.

Reliability: It provide durable state storage designed to be highly available and fault-tolerant.

Consistency: It use optimistic concurrency control to ensure that the state of an entity is consistent and that updates are applied in the correct order.

Encapsulation: Entity Functions allow you to encapsulate the logic and state of an entity in a single class, making it easier to maintain and evolve.

Reusability: Entity Functions can be used across multiple workflows and orchestrations, providing a way to reuse familiar stateful entities across different parts of an application.

When you use Entity operations in your function, you can use the following types of operations:

Entity Id: Your target entity

Operation Name: Name of the operation to be performed on the entity

Operation Input (Optional): Input parameter for the operation

Scheduled Time (Optional): Delivery time of the operation

Durable Entities in .NET

If you are using .NET for Durable Entity Functions, there are two ways you can define your durable entities: class-based and function-based. The class-based syntax usually describe entities and its operations as classes or methods which is is more flexible and readable, especially if you want to use interaces. On the other hand, function-based syntax allows you to get precision control over your entity functions such as how entity state is managed or the entity operations are being dispatched.

Limitations of Entity Functions

Entity functions and related functionality in Durable Functions are only available in version 2.0 and higher. They are currently supported in .NET, JavaScript, and Python. As of date of writing, the feature is not available in Java or Powershell yet.

Overall, Entity Functions are a powerful feature of Azure Durable Functions that provide a way to manage stateful entities in a distributed, scalable, and reliable manner. They allow you to encapsulate the state and behavior of an entity in a single class and provide a way to ensure consistency and reliability when multiple Durable Functions access the same entity simultaneously.

If you want to see the entire source code of these durable function types examples, please free to clone my GitHub repository: Azure Durable Functions (Function Chaining Example) in C# .NET (Starter Template). You may use it as a template to get started with your own durable functions or as a reference to this section of the book.

Orchestration Trigger kickstarts Durable Functions

So how do Durable Functions get triggered? Durable Functions have trigger bindings which helps in the execution of the functions for orchestration, activity and entity. The Orchestrator trigger is designed to execute if a new instance of orchestration is being scheduled. Or when the current instance of the orchestrations receives a new event. Such event can be coming from external clients, HTTP trigger, durable timer events, etc.

The Orchestrator trigger configuration is unique and can be customized depending on the the type of programming languages you are using to code your function app. For example, authoring functions in .NET uses the class OrchestrationTriggerAttribute. If you are authoring it in Powershell, Javascript, or Python, you need to define the configuration for this trigger on the functions.json file, as shown in Example 3-5.

Example 3-4. Example format of the functions.json file which describes the configuration for a durable function app. The property orchestration is optional.

{
    "name": "<Name of input parameter in function signature>",
    "orchestration": "<Optional - name of the orchestration>",
    "type": "orchestrationTrigger",
    "direction": "in"
}

The Orchestrator is Deterministic

Of all the Durable Functions types, the orchestrator functions play an important role in keeping your workflow running smoothly. Event sourcing is used in order to maintain the state and reliability of task execution. The orchestrator need to be designed in a deterministic way because the workflow you write in the code will be replayed many times. It expects the same result each replay. For this reason there are some documented and recommended code constraints to follow when designing the workflow in the orchestrator.

Table 3-3 lists some of the code constraints, or dos and don’ts, that you need to know when designing and writing your workflow in the orchestrator function.

Table 3-3. The DOs and DONTs of Writing a Workflow Orchestration in the Orchestrator Function

DONTs
DOs - What you can do instead

Avoid generating GUIDs or random numbers

Make use of the NewGuid() method within the context

Don’t access configurations and data

Pass data stores or configuration into activity functions

Never create infinite loops in the orchestrator

CreateAsNew() method in the Durable Orchestration context can be used instead

Prevent having blocking APIs and threads (Sleep, Run, Delay)

Use the DurableOrchestrationContext to control the state of your tasks

Never use standard CurrentDateTime() method ex. in .NET

Use CurrentUtcDateTime() method in context instead

There are other important things to consider when you design your stateful orchestration with Azure Durable Functions that fall outside the scope of this book but may be useful for your particular use case. Learn more about the other common orchestrator function code constraints in the Microsoft documentation.

Durable Function Types - Stateful or Stateless?

One of the features that makes Durable Functions effective and useful is its capability to keep the workflow or orchestration of our activity functions stateful. Redhat has a good cloud native article explaining about what is the difference between stateful vs. stateless and the Microsoft article Affairs of State: Serverless and Stateless Code Execution with Azure Functions is also a worth checking out especially for design considerations and best practices for development for Azure Functions in serverless architecture.

Figure 3-10. The different state for each of the Durable Function Types

Figure 3-10 gives us an illustration what state each durable function type is. Client and activity functions are like the standard Azure Functions that are stateless. On the other hand, the Orchestrator and the Entity functions are stateful.

Application Patterns for Serverless Stateful Workflows

One of the key features that makes it a great serverless compute service is that it enables us to write stateful workflows in serverless architecture or environment. Stateful workflows and orchestrations are special in a way that it helps us solve some of the complex problems in our applications. Normally, complex orchestration use case scenarios are difficult to handle by just using normal functions that loses its state (stateless)³.

There are six known application patterns that have been documented by Microsoft for Azure Durable Functions. These patterns are useful recommendation in solving some of the common complex use case scenarios in the serverless architecture that we will explore next.

Function Chaining

In the function chaining pattern shown in Example 3-6, a sequence of functions executes in a specific order like a chain. The output of one function is applied as the input of the next function.

Figure 3-11. An example of Function Chaining pattern

Example 3-5. A code block example of Function Chaining pattern from reference source Microsoft Documentation: Azure Durable Functions

[FunctionName("FunctionChainingExample")]
public static async Task<object> Run([OrchestrationTrigger] IDurableOrchestrationContext context)
{
    try
    {
        var result1 = await context.CallActivityAsync<object>("MyFunction1", null);
        var result2 = await context.CallActivityAsync<object>("MyFunction2", result1);
        var finalresult = await context.CallActivityAsync<object>("MyFunction3", result2);
        return  await context.CallActivityAsync<object>("MyFunction4",finalresult);
    }
    catch (Exception)
    {
        // Error handling, activities or retry-functions to handle exception
    }
}

Function Chaining is a Durable Functions pattern allowing you to execute a series of functions in a specific order. In this pattern, each function in the chain depends on the output of the previous function, forming a chain of process calls that pass data from one function to the next.

This pattern is useful when you need to break down a complex task into smaller, independent steps and ensure that the steps are executed in the correct order. This pattern can be used in various scenarios, such a processing a batch of files, performing a series of calculations, sending a sequence of messages, and other tasks that requires sequence processing.

This pattern is achieved b the implementation of an orchestration function that calls a series of activity functions. The orchestration function is responsible for managing the state of the workflow and calling the activity functions in the correct order.

The basic workflow of the Function Chaining pattern in Azure Durable Functions is as follows and illustrated in the code example:

When an activity function is called, it executes as a separate Azure Function and returns its result to the orchestrator function. If the activity function fails or times out, the orchestrator function can handle the error and retry the activity function if necessary.

The Function Chaining pattern in Durable Functions is a method to break down difficult tasks into smaller, independent steps and ensure that the steps are executed in the correct order. It provides a way to manage the state of the workflow and handle errors and retries efficiently.

Fan Out / Fan In

Fan-out/Fan-in is a pattern in Azure Durable Functions that lets you execute multiple functions at the same time and aggregate the result. In this pattern, various functions process a single input in parallel, and the resulting output is combined. You code and author workflow to execute multiple functions in parallel. Then you wait for all functions to finish, as shown in Example 3-7. Usually the aggregation work is done on the results that are returned from the functions, as illustrated in Figure 3-12.

Figure 3-12. An example of Fan-out/Fan-in Pattern

This orchestration pattern is advantageous when a large quantity of data must be processed or multiple independent tasks must be completed concurrently. This pattern is applicable in various situations, including data processing, batch processing, and parallelizing compute-intensive tasks in various use cases in some applications.

Example 3-6. A code block example of Fan-out/Fan-in pattern from reference source Microsoft Documentation: Azure Durable Functions

[FunctionName("FanOutFanInExample")]
public static async Task Run([OrchestrationTrigger] IDurableOrchestrationContext context)
{
    var parallelTasks = new List<Task<int>>();

    // Get a list of N work items to process in parallel.
    object[] workBatch = await context.CallActivityAsync<object[]>("F1", null);
    for (int i = 0; i < workBatch.Length; i++)
    {
        Task<int> task = context.CallActivityAsync<int>("F2", workBatch[i]);
        parallelTasks.Add(task);
    }

    await Task.WhenAll(parallelTasks);

    // Aggregate all N outputs and send the result to F3.
    int sum = parallelTasks.Sum(t => t.Result);
    await context.CallActivityAsync("F3", sum);
}

In this pattern, the orchestration function is responsible for administering the workflow’s state and coordinating the execution of activity functions in parallel.

A workflow starts with an orchestration function that initializes the workflow’s state and prepares input data for parallel processing. Using the ‘CallActivityAsync’ method, the orchestration function calls multiple activity functions in parallel and transmits a partition of the input data to each activity function. Each activity function executes the assigned task autonomously and returns the result to the orchestrator function. The orchestrator function awaits completing all activity functions using the ‘Task.WhenAll’ method. The orchestrator function returns a single output containing the outcomes of all activity functions.

To implement this pattern in Durable Functions, you must define the activity functions as distinct and decorate them with the '[FunctionName]' attribute. You then contact them in a loop from the orchestration function using the ‘CallActivityAsync’ method, passing each activity function a partition of the input data.

When an activity function is invoked or triggered, it runs as its own Azure Function and returns its result to the orchestrator function. The orchestrator function can manage the error and retry the activity function if necessary if the activity function fails or times out.

Overall, this pattern is an effective method that executes multiple functions in parallel and aggregates their results into a single output. It provides a way to manage a workflow’s state while effectively handling errors and retries.

Async HTTP APIs

If you want to solve the problems related to state coordination of long-running operations with external clients or APIs, then Async HTTP APIs pattern might be useful in solving it.

A common way to implement Async HTTP API pattern is by having the long-running action triggered by an HTTP call, and then redirecting the client to a status endpoint that they can poll to learn when the operation completes. Figure 3-13 shows a simple illustration on how the flow of it starting, doing the work and getting the status over HTTP.

Note

Durable Functions provide built-in APIs that simplify the code we write for interacting with long-running function executions.

Figure 3-13. An example of Async HTTP APIs Pattern

Example 3-8, shows a simple way how we can use Async HTTP APIs pattern to make a HTTP request to any URL or endpoint.

Example 3-7. A code block example of Async HTTP API pattern checking website status and content via HTTP

[FunctionName("CheckWebsiteContentFunction")]
public static async Task CheckWebsite([OrchestrationTrigger]
                    IDurableOrchestrationContext context, ILogger log)
{
    try
    {
       Uri jonahsWebsite = new Uri("https://jonahandersson.tech");

       //Make an HTTP request ex. GET or POST to specific endpoint or URL
       DurableHttpResponse httpResponse = await context.CallHttpAsync(HttpMethod.Get,jonahsWebsite);

       if((int))httpResponse.StatusCode >= 500 && (int)httpResponse.StatusCode < 600)
            throw new HttpRequestException("Hey, there is a server error!");
       else
            log.LogInformation($"Completed");
            log.LogInformation($"Response Code: {httpResponse.StatusCode}");
            log.LogInformation($"Content: {httpResponse.Content}");
    }
    catch (Exception)
    {
        // Error handling, activities or functions to handle exception.
    }
}

Finally, if you want to learn more about how you can use Durable Functions Async HTTP APIs and exposing asynchronous, long-running processes over HTTP, check out this Microsoft documentation about its HTTP Features.

Monitor Pattern

In context with a workflow, the monitoring pattern relates to an adaptable iterative procedure, such as polling until specific requirements are satisfied. A simple case, such as a recurring cleanup task, can be handled by a regular timer trigger; however, because its interval is fixed, it makes managing the instance lifetimes more difficult.

The benefit of Durable Functions makes it possible to construct numerous monitor processes from a single orchestration and have adjustable recurrence intervals and job lifespan management.

An example use case where monitor pattern can be useful is when you have to run monitoring job processes, as illustrated in Example 3-9.

Example 3-8. A code block example of recurring monitoring job workflow using Monitor pattern

[FunctionName("MonitorCleanUpJobStatus")]
public static async Task Run([OrchestrationTrigger] IDurableOrchestrationContext context)
{
    int cleanUpJobId = context.GetInput<int>();
    int jobPollingInterval = GetPollingInterval();
    DateTime expirationTime = GetExpiryTime();
    string emailAddress = GetEmailAddress();

    while (context.CurrentUtcDateTime < expirationTime)
    {
        // Monitor job status by job id.
        var status = await context.CallActivityAsync<string>( "GetJobCleanUp", cleanUpJobId);
        if (status == "Completed")
        {
            // Perform an action when a condition is met.
            await context.CallActivityAsync("SendMonitoringAlert", emailAddress);
            break;
        }

        // Orchestration sleeps until this time.
        var nextMonitorCheck = context.CurrentUtcDateTime.AddSeconds(jobPollingInterval);
        await context.CreateTimer(nextMonitorCheck, CancellationToken.None);
    }

    // Perform more work here, or let the orchestration end.
}

So, the monitor pattern can be useful in implementing tasks or monitoring jobs that are recurring and needs to be stateful. You can also combine it with durable timer to control the time and workflow based how you prefer it to be.

Human Interaction

There are some processes and tasks that requires some kind of human interaction. The tricky thing about involving us humans in an automated process is that we are not always as highly available 24/7 and responsive like computer systems or cloud systems. For this major reason, developing solutions for the automation of processes can be beneficial. Often timers and programming logic are used to do this, as shown on example of approval work flow on Figure 3-14 and Example 3-15.

Figure 3-14. An example of Human Interaction Pattern

Example 3-9. A code block example of a simple manager approval workflow coded in C# using Human Interaction pattern

[FunctionName("ApprovalFromManagerWorkflow")]
public static async Task Run([OrchestrationTrigger] IDurableOrchestrationContext context)
{
    await context.CallActivityAsync("RequestApprovalFromManager", null);
    using (var timeout = new CancellationTokenSource())
    {
        DateTime dueTime = context.CurrentUtcDateTime.AddHours(72);
        Task durableTimeout = context.CreateTimer(dueTime, timeout.Token);

        Task<bool> humanInteractionEvent = context.WaitForExternalEvent<bool>("ManagerApprovalEvent");
        if (humanInteractionEvent == await Task.WhenAny(humanInteractionEvent, durableTimeout))
        {
            timeout.Cancel();
            await context.CallActivityAsync("ProcessRequest", humanInteractionEvent.Result);
        }
        else
        {
            //If there is no approval or it timed out
            await context.CallActivityAsync("EscalateToAnotherPerson", null);
        }
    }
}

Aggregator Pattern

If you want to collect or aggregate event data over a specific period of time into a single, transmittable entity, then the aggregator design pattern is ideal to be used in your durable orchestration. The event data that are being collected and aggregated can be coming from several and different multiple resources which can also be delivered into batches or groups. These event data may be scattered over time. The aggregator is designed to take possible logical action when the event data comes and those external services or clients might need to query these event data that are being aggregated.

The aggregator pattern is not easy to implement in normal functions that are not stateful because of the big challenge of concurrency problems. In normal stateless functions, you need to handle the concurrent multiple threads with same data and also make sure that the aggregator runs on a VM at a time.

There are code examples to implement the aggregator pattern for stateful entities.

These different application patterns of Durable Functions are useful in different scenarios depending on the type of complex problems or use case scenarios you are trying to solve in your serverless or backend part of your applications. If you want to learn more about real examples, there are a few publications about the use of serverless and Azure Durable Functions on the following papers.

Durable Functions: Semantics for Stateful Serverless

Serverless Workflows with Durable Functions and Netherite

Container Services in Azure

One of the great ways to modernize applications for the cloud platform like Microsoft Azure is to build and develop them using containers. One of the major benefits of using containers is Azure is cost-savings for organizations who wants to lift and shift their existing applications to the cloud without making making huge changes. By containerizing existing applications or building them into microservices can help deliver value to the business and application users.

Applications that are containerized have great capabilities like modern tools for development and CI/CD with fully managed container management using the services like Azure Kubernetes. Containers can also be fully controlled and integrated with Azure Active Directory for user access management and security.

Azure Containers and Azure Kubernetes Service

Container Instances and Azure Kubernetes Service are Azure compute resources that you can use to deploy and manage containers. Containers are lightweight, virtualized application environments. They’re designed to be quickly created, scaled out, and stopped dynamically. You can run multiple instances of a containerized application on a single host machine.

Azure Container Registry

Azure Container Registry (ACR) is a cloud-based managed service that enables users to store, administer, and deploy container images with Azure Kubernetes Service (AKS) and other container orchestrators. ACR serves as a private registry that enables users to store and administer Docker images in a secure and scalable manner.

ACR enables users to rapidly deploy and administer containers in the cloud using the Docker CLI tools they are already familiar with on their local machines. ACR supports Windows and Linux containers, enabling integration with other Azure services, including Azure DevOps, Azure Container Instances, and Azure Kubernetes Service.

You can also use Azure Container Registry Tasks (ACR Tasks) to build your container images in Azure on-demand, or automate builds triggered by source code updates, updates to a container’s base image, or timers.

ACR offers a variety of features, such as role-based access management (RBAC), geo-replication, container image scans, and integration with Azure AD. RBAC permits users to manage access to their container images by granting permissions to specified users and groups. Geo-replication allows users to replicate container images across numerous Azure regions for increased availability and resiliency. Users can scan their container images for vulnerabilities and security issues using image scanning.

Getting started in creating Azure Container Registry

You can create Azure Container Registry in different ways using the Azure CLI, Azure Portal, PowerShell, Bicep, and ARM templates.

ACR is a powerful tool that enables users to securely and scalably store, administer, and deploy images of containers in the cloud.

Azure Container Instance

Deploying applications and systems in containers is one of the common app modernization technologies today. Many organizations are considering containerization for many reasons including the benefits of convenient packaging and deployment of applications into containers to the cloud.

Figure 3-15. An example overview of Azure Container Instance

Azure Container Instances (ACI) allows us to run containers in Azure instead of running and managing virtual machines for our applications, as illustrated on Figure 3-15. ACI allows you to run and isolate applications and implement task jobs automation.

Azure Container Apps

Azure Container Apps (ACA) is compute service that is designed for developing and deploying applications and microservices using serverless containers. It has great features including containerized apps without managing complex infrastructure and also support for built-in authentication for security.

Although it is not required to use the built-in authentication and authorization in Azure Container Apps, it useful because you make use of the variety of common authentication and identity providers including third-party providers e.g., Microsoft Identity Platform, Azure Active Directory, Google, Twitter, GitHub, etc.

ACA combines the benefits of serverless and PaaS that provides a platform for multiple container applications in a serverless environment as shown in Figure 3-16. This container solution in Azure as application lifecycle in three different phases e.g., Deployment, Update, and Deactivate based on revisions.

When do Azure Container Apps Shutdown?

Aside from the different phases of the application lifecycle mentioned, the Azure Container Apps also has a shutdown phase. This phase is triggered when the container app is being deactivated, delated or being scaled in for updates.

Figure 3-16. An example illustration of the environment of Azure Container Apps with containers apps or microservices.

There are different container services in Azure but the following benefits and uses shown on Table 3-3 defines an overview of how it differs from the others.

Table 3-4. Common Uses and Benefits of Azure Container Apps

Uses and Benefits
Description

Running applications in Microservices

Develop, deploy and manage your microservices applications in serverless containers with Distributed Application Runtime(Dapr) integration

Deploying API endpoints

The HTTP traffic can be split into different versions of an application with autoscaling features based on concurrent HTTP traffic or requests

Hosting applications with background processing

Run stateful and long-running background jobs or tasks with autoscaling capabilities based on CPU or memory usage

Managing even-driven processing

Manage and auto scale event-driven processes

Implementing Azure Container Apps mean that you can build containerized applications code using any preferred programming language or frameworks, build microservices, and have the option to use it with technologies like Distributed Application Runtime(Dapr), Kubernetes Event-Driven Autoscaling (KEDA) and envoy.

Figure 3-17. An example illustration of Azure Container Apps

Figure 3-17 gives an overview of Azure Container Apps and its features along with integration options for Dapr, KEDA, Envoy and the Azure Kubernetes Service (AKS).

Learn more about KEDA by watching this video KEDA: Event Driven and Serverless Containers in Kubernetes - Jeff Hollan, Microsoft, you may also go through the lessons on the Microsoft Learning Path, Scale container applications in Azure Kubernetes Services (AKS) using KEDA

If you are also curious about the official KEDA documentation, visit the official GitHub page of KEDA for code examples and also check out KEDA scalers that helps you detect if a deployment should be disabled or enabled.

Note

KEDA is a Kubernetes-based Event Driven Autoscaler. KEDA allows your to manage and control the scaling of any container in Kubernetes according to the number of events that needs to be processed. Using Azure Container Apps with KEDA (Kubernetes Event-Driven Autoscaler) allows you to automatically scale your applications down to 0 based on HTTP traffic or requests, CPU or memory or any of KEDA’s event-driven scalers like Azure Service Bus, Azure Monitor, Azure Event Hub, SQL DB, Redis and more).

Aside from making use of KEDA, you can also deploy an Azure Container App from Azure Container Registry (ACR) using Azure CLI and set up the CI/CD using Azure DevOps Pipelines.

Furthermore, ACA is considered top-level Azure resources which means that you can develop it as Infrastructure as Code (IaC) methods using ARM templates, Azure Bicep, Azure CLI, Powershell and other external tools like Terraform and ARM resources are top-level Azure resources.

Azure Kubernetes Services

Kubernetes (K8s) is open-source orchestration software for deploying, managing, and scaling containers. Azure Kubernetes Service (AKS) is ideal if you have use case scenarios where you need full container orchestration. AKS also includes that feature of coordinated application upgrades, automatic scaling, and a feature to perform service discovery across multiple containers.

AKS is a managed Kubernetes container orchestration service. Kubernetes is container management system that is open-source. It helps automates containerized applications’ deployment, scaling, and administration. It provides a highly available, scalable, and secure platform for deploying and managing Kubernetes-based containerized applications. It offers automated upgrades, self-healing, monitoring, and scalability, making it an ideal solution for deploying and managing containerized cloud applications.

Among the essential uses of AKS in cloud development are:

Container Orchestration

AKS simplifies the deployment and management of containerized applications using Kubernetes for container orchestration. It provides a platform that automates container deployment, scaling, and administration, making large-scale container deployments simple to manage.

Hybrid Containers

AKS can deploy containerized applications on-premises, in the cloud, or in a hybrid infrastructure. This facilitates the creation of a deployment environment that is consistent across multiple locations.

Container Security

AKS offers a secure platform for operating containerized applications, including network security, role-based access control, and container isolation. This ensures that applications are compliant with industry standards and safe.

Support for CI/CD and DevOps Lifecycle

AKS can be integrated with Azure DevOps, GitHub, and Jenkins, among other DevOps tools, to automate the deployment, testing, and monitoring of containerized applications. This allows developers to concentrate on application development while AKS manages the underlying infrastructure.

Furthermore, among the advantages of utilizing AKS is enabling developers to rapidly and easily scale containerized applications based on demand. This ensures that applications are accessible and responsive to user requests at all times.

Figure 3-18 shows a simple example of a use case of AKS in a microservices architecture.

Figure 3-18. An example reference architecture of microservices architecture on Azure Kubernetes Service.

AKS can be a cost-effective method for running containerized applications, removing the need to administer and maintain the underlying infrastructure if you want to implement container technologies on the cloud in the highly available platform for operating containerized applications, complete with automatic failover and self-healing capabilities.

Generally, AKS is a fully managed Kubernetes container orchestration service in Azure that helps with the deployment and management of containerized cloud applications.

Microsoft’s Free Learning Books for Kubernetes

Microsoft Azure has free learning resources that you can check out to get started about Azure Kubernetes, Distributed Systems, and Cloud Native Computing architecture.

AKS helps container development simple because of container orchestration capabilities. It also provide developers with the ease of containers deployment and management. Also, one of the good features of is the node auto-repair.

Quantum Computing in Azure - Making Sense of it

Quantum computing is new to many of us. It is promising when the information we read about quantum computing is anywhere. It seems interesting this technology might be the solution to our most significant challenges in the world today - in the field of agriculture, health, energy, material science, etc.⁴ Quantum computers are controllable quantum mechanical devices that exploit the properties of quantum physics to perform computations.⁵

Quantum computing provides the exponential advantages when it comes to increasing the speed of some computational tasks. These advantages are possible because of quantum mechanics: superposition, interference, and entanglement. The difference between standard computers and quantum computers is the applications in quantum computers are probabilistic. In comparison, the computers we use daily are usually deterministic. Each possible result produced by quantum algorithms has an associated probability amplitude.⁶.

There’s a lot that can be said on this topic, and if you’d like to learn more, you can read more on Quantum computing history and background.

Uses of Quantum Computing

Quantum computers are useful in solving problems that require complex calculation with variations of possible combinations. For example using quantum computing in the fields of cryptography, mechanical systems simulation, machine learning, and algorithms.

Azure Quantum

Azure Quantum is the quantum cloud platform of Microsoft. It is cloud service with a diverse set of quantum solutions and technologies used to solve some quantum computing related problems. Even though quantum computing is a new technology, you can prepare for the future of it, by starting to write your code at the algorithm level.

Azure Quantum Development Kit

Azure Quantum Development Kit (Azure QDK) used for development with Azure Quantum using Q#, our quantum-focused programming language and Azure Quantum, our quantum cloud platform. Build and run Q# programs on quantum hardware or formulate solutions that execute optimization solvers running on classical hardware on Azure.

Getting Started with Azure QDK and Q#

Quantum computers are useful in solving problems that require complex calculation with variations of possible combinations. For example using quantum computing in the fields of cryptography, mechanical systems simulation, machine learning, and algorithms. To try an example yourself, check out this Quantum Random Number Generator in Q#

Azure Quantum seems promising technology; however, it is not widely used by general public and developers yet. I have attended one of the Azure Quantum Developer Workshops in the past, and I am very excited to see how it will be used in the future. I created my first Q# program in 2021 and there are interesting concepts to learn about it.

If you want to try it out to explore, I recommend Microsoft’s Quantum Learning Resources like the Quantum Computing Foundations Learning Path and join the Q# Community.

Learn By Doing (Try it out!)

Aside from the supplementary learning hands-on lab repository for the topics in the this chapter, the following quick-start tutorials are most recommended as they are updated based on Microsoft’s technical updates for the service. This section is recommended and optional.

Microsoft Quickstart Tutorial for creating a Windows VM in Azure Portal

Microsoft Quickstart Tutorial for creating a Linux VM in Azure Portal

Microsoft Quickstart Tutorial hosting a web application using Azure App Service

Microsoft Quickstart Tutorial on how to deploy and run a containerized web app with Azure App Service

Choose your desired programming language by following through the serverless lessons at Azure Functions University on GitHub

Microsoft Quickstart Tutorial for deploying a Dapr Application to Azure Container Apps using Azure CLI

Microsoft Quickstart Tutorial for deploying a Dapr Application to Azure Container Apps using Azure CLI

Microsoft’s Tutorial for deploying a Dapr Application to Azure Container Apps using an ARM or Bicep template

Quickstart: Solve an optimization problem in Azure Quantum

Summary

In this chapter, we explored Azure Compute, a widely-used category of services in Azure that enables the creation of modern applications and systems. Azure Compute supports various architectures such as serverless, microservices, and containers, and provides benefits such as auto-scaling, consumption-based pricing, and enhanced developer productivity. We also learned about Azure Durable Functions, which allows the creation of serverless stateful workflows and orchestrations through code while adhering to certain authoring and coding constraints.

Azure App Service and Azure Static Web Apps were discussed as tools for empowering modern app development. Azure Functions and Durable Functions were highlighted as powerful backend services and function apps with event-driven features for stateful workflow orchestrations. Azure Kubernetes, Azure Container Registry, and Azure Container Apps were introduced as solutions for creating container apps with orchestration, scaling, and load-balancing capabilities. Additionally, you learned about Azure Quantum, a service and Quantum SDK for creating cloud solutions that leverage quantum computing to solve complex problems.

Overall, Azure Compute provides a robust suite of tools for developers, teams, and organizations to build and deploy more scalable, efficient, and secure applications and systems.

Finally, all these compute services are available in Azure so that developers, teams, and organizations can build and deploy better applications and systems that are more scalable, efficient, and secure.

Check Your Knowledge

What is the difference between Azure VMs from Azure VM Scale Sets?

What are Azure Spot VMs?

Why do you think Azure Container Apps are the best way to deploy microservices?

What Azure Compute service would you use if you want to write stateful serverless workflows?

What programming language and developments tools would you primarily need for Azure Quantum?

To find the answers to the Check Your Knowledge, turn to back of the book in the Appendix

Recommended Resources

AzureCharts.com

Choose an Azure compute service for your application

Microsoft Azure Functions Documentation

Azure Durable Functions Documentation

Developer’s Guide to Durable Entities in .NET

Microsoft Documentation of Azure Container Registry

Azure Container Apps Official Documentation

Azure App Service Microsoft Documentation

Azure Quantum Documentation (Preview)

Learn the Basic of Kubernetes

Core concepts for Azure Kubernetes Service (AKS)

Microsoft Azure Architecture Center - Design Patterns

¹ RedHat.com, “What is virtualization? “,https://www.redhat.com/en/topics/virtualization/what-is-virtualization

² W3.org, “CORS Enabled? “,https://www.w3.org/wiki/CORS_Enabled

³ Mikhail Shilkov. 2018, “Making Sense of Azure Durable Functions”, https://mikhail.io/2018/12/making-sense-of-azure-durable-functions/

⁴ Rachel Gordon. 2022, “A new language for quantum computing”, MIT News, https://news.mit.edu/2022/new-language-quantum-computing-twist-0124

⁵ Michael Tabb, Andrea Gawrylewski, and Jeffery DelViscio. 2021, “How Does a Quantum Computer Work?”, Scientific American, https://www.scientificamerican.com/video/how-does-a-quantum-computer-work/

⁶ IBM.com, “What is quantum computing? “,https://www.ibm.com/topics/quantum-computing

Chapter 4. Microsoft Azure Cloud Networking

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 4th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at jleonard@oreilly.com.

Azure networking is essential to any public cloud, it allows us to bring both on-premise and cloud networks together. We can scale resources to meet our demands as well as protect our infrastructure; this gives us the flexibility to change to meet any resource demand that our organization or customer may need. With Azure networking, we get compliance and security. It also saves on costs and time which gives us the flexibility to adapt and meet our demands, both current and future.

Ryan O’Connell, IT Solutions Architect, Microsoft Azure MVP, Microsoft Certified Trainer, IT Manager, Blogger at RockITWorks

Now that we’ve learned about the common modern Azure compute services and their benefits for application development in Microsoft Azure, we will deep dive into the concepts you need to learn to implement these compute services with the known networking services in Azure. By the end of this chapter, you will have learned how you can use the appropriate networking service with existing applications. These will also help in planning networking and hybrid solutions in Azure.

Azure Networking

Azure Networking is a category of services in Microsoft Azure that provides fully- managed and scalable networking and connectivity options like making a connection between your on-premise data center to the cloud. With networking services in Azure, you can also build secure virtual network infrastructure, manage your application’s network traffic and protect them against DDoS attacks. Networking resources in Azure can also be used to enable secure remote access to internal resources within your organizations and globally route your network connectivity with features of monitoring and security.

Figure 4-1 shows us an overview of some of the most commonly used networking and connectivity resources in Microsoft Azure. These services help in integrate your on-premise data center with Azure, load balancing your network traffic, and protecting your network from DDoS attacks.

Figure 4-1. An overview of the common networking, connectivity and network security services in Azure

Using your Azure subscription, while logged in, you can view all of the Azure resources you can create on the networking category on the Azure Portal. You just need to click Create a resource and select the Networking category on the Azure Marketplace, and there you can explore the rest of the networking resources by Microsoft and Microsoft Partners.

By making use of the Azure Marketplace you will be able to see a list of different types of Azure resources, apps and services for different categories and deployment models.

Azure Networking Services Categories

There are many networking services to choose from in Azure and they are categorized into groups according to their purpose.

Services for Connectivity

The services under this group category are the Azure networking resources that you can create and build for connectivity-related solutions in the cloud. For example, if you want to connect your Azure resources to your on-premise resources, you can use them such as Azure Virtual Network (Azure VNet), Express‐ Route, Virtual WAN, Virtual network NAT Gateway, VNet Peering service, VPN Gateway, Azure Bastion, and Azure DNS.

Services for Application Protection

The networking services under this group category are intended for networking resources that will help secure and protect your applications or systems in Azure. For example, you can implement networking services like Azure Load Balancer, Firewall, VNet Endpoints, Private Link, and DDoS protection.

Services for Application Delivery

The services under this category are ideal in use-case scenarios you need for application delivery. Such networking resources include Azure Content Delivery Network (Azure CDN) for content delivery, global routing web traffic using Azure Front Door Service, load balancing your traffic across Azure regions globally using the Azure Traffic Manager, and other resources like Application Gateway, and Internet Analyzer.

Services for Network Monitoring

Monitor your network resources using any or a combination of these networking services in Azure - Network Watcher, ExpressRoute.Monitor, Azure Monitor, or VNet Terminal Access Point (TAP).

We’ll explore each of these categories in greater detail throughout the rest of the chapter to give a better understanding of the different networking services for different use cases and scenarios.

Azure Networking Services for Connectivity

Azure provides a network infrastructure that is robust, fully managed, and dynamic to support complex network architecture. These network solutions vary from creating public access to applications network security and making hybrid connections between infrastructure on-premises and on the cloud.

Azure Virtual Network

Azure Virtual Network (Azure VNet) plays an important role in building the net‐ works within the Azure infrastructure. It is a fundamental block of keeping your Azure resources in a private network where you can securely manage and connect to other external networks (public and on-premises) over the Internet.

Azure VNet goes beyond the usual on-premise and traditional networks that we know. Aside from the benefits of isolation, high availability, and scalability, Azure VNet helps secure your Azure resources by allowing you to administrate, filter, or route network traffic based on your preference.

For example, Azure VMs are basically connected or attached to an Azure VNet through virtual network interface cards (VNICs), as shown in Figure 4-2, where an Azure VM is attached to three NICs: Default, NIC1, and NIC2)

Figure 4-2. An example of a Three-Tier VNet, an Azure VM with several NICs

Azure resources need to securely communicate with each other internally in its private network, over the internet, and also networks on on-premise infrastructure. Azure VNet makes this possible for these. Let’s take a closer look at these forms of communication.

Internet Communication

Azure VNet, by default is capable of communicating outbound to the Internet. If you want to communicate to an Azure resource inbound, you may do that using an Azure Public IP address or Azure Load Balancer that is set public.

Communication and Connection with Azure resources

Azure resources can securely communicate through the virtual network, VNet Peering and by extending virtual network service endpoints. For example, it is possible to deploy some dedicated Azure resources like AKS, Azure Batch, Azure SQL Manage Instance, Azure AD Domain Services, Azure Container Instance (ACI), Azure Functions, Azure App Service Environments, Azure VM Scale Sets within the same virtual network.

Network traffic routing and filtering

The network traffic can be filtered between subnets in several ways. Network security groups (NSG) and application security groups can be used to control security rules (inbound and outbound) to control and filter network traffic. Another good option is the make use of a network virtual appliance virtual machine where you can configure your firewall settings, and network rules and optimize your WAN. On Azure Marketplace, you will find some of networking appliance managers available for external services or within Microsoft.

Azure VNet Peering

Azure Virtual Network Peering (VNet Peering) allows you to connect several virtual networks in Azure. The connectivity and traffic between the virtual machines in the peered virtual networks uses Microsoft’s infrastructure on a secure private network. Virtual networks that are peered can directly share and connect with the resources located in either virtual networks.

Currently, Azure supports both Virtual network peering and Global network peering. The difference between these two is that Global network peering connects virtual networks across Azure regions while Virtual network peering connects virtual networks within the same Azure region.

Figure 4-3 shows us an example illustration and overview of how VNet Peering works between two virtual networks, VNet A and VNet B which has a several connections to other networking resources.

Figure 4-3. An example of a VNet Peering in Azure

Both virtual network peering and global virtual network peering support gateway transit, which is a property in peering that allows a virtual network to make use of the peered virtual network’s VPN gateway for the cross-premises connectivity or VNet-to-VNet connectivity. You can configure Configure VPN gateway transit for VNet peering through Azure Portal, Powershell, ARM template and Azure CLI.

Troubleshooting VNet Peering Issues

Azure VNet Peering issues can occur; therefore, troubleshooting is necessary to understand the root cause of the issue. One of the common troubleshooting questions to consider is whether the virtual networks are in the same subscription or not and if they are in the same region or a different region. For more details about different types of problems with VNet peering, please refer to Microsoft’s documentation on Troubleshooting VNet Peering issues.

On a final note, VNet peering uses IP addresses. It is good to know that there are two types of IP addresses that are you used in Azure - Public IP and Private IP The Private IP addresses are used for the connectivity and Azure resources communication within the same Resource Group. On the other hand, Public IP addresses are used to allow Internet resources to communicate inbound to Azure Azure resources. Using this type of IP address enables communication of Azure resources and public-facing Azure services over the Internet.

Learn more about IP services and some recommended best practices for Azure Virtual Network.

Azure Virtual Wide Area Network

Azure Virtual WAN is a managed networking service and unified framework used for networking, security, and routing functionalities. The Azure Global Network is what enables Azure Virtual WAN to be available. It includes Site-to-site, Point-to-site VPN connectivity, ExpressRoute, etc.

Virtual WAN helps organizations or business units to connect to the Internet and other Azure resources. For example, networking and remote user connectivity, which is effective and useful for those who prefer to work from home or anywhere they want.

The following are some of the known features of Azure Virtual WAN:

Branch connectivity (via connectivity automation from Virtual WAN Partner devices such as SD-WAN or VPN CPE).

Site-to-site VPN connectivity.

Point-to-site VPN connectivity for remote users

Private connectivity using ExpressRoute

Intra-cloud connectivity for virtual networks

Inter-connectivity using VPN ExpressRoute

Routing, Azure Firewall, and encryption for private connectivity

Azure Virtual WAN also has benefits like integrated connectivity solutions in the hub and spoke, automated spoke configuration, and troubleshooting and monitoring tools. Also, when configuring it, you would need Virtual WAN resources such as Virtual WAN, Virtual Hub, Hub virtual network connection, Hub-to-hub connection, hub route table, and site resource

Figure 4-4 shows an example of Azure Virtual WAN being used and implemented for remote connectivity.

Figure 4-4. An overview Azure Virtual WAN,

Migrating to Azure Virtual WAN

Since each migration is unique, we won’t cover this in detail. However, there are options and resources available to help you navigate your migration to Azure Virtual WAN.

Azure Virtual WAN is a broad and complex topic itself. If you want to deep dive into it, I recommend reading Microsoft’s Virtual WAN Documentation.

Azure Express Route

Azure Express Route allows you to expand networks that are on on-premises into Microsoft’s cloud infrastructure using a connectivity provider over a private connec‐ tion. Basically, this networking service gives the capability of connecting your on-premise networks with Azure, Microsoft’s cloud. This connection between an on-premise network and Azure can be initiated using an any-to-any (IPVPN) network with Layer 3 connectivity, which enables you to interconnect with Microsoft’s cloud to your own on-premise WAN or data center.

The connection in Azure Express Route is private, and the traffic with this connection does not go over the Internet. This means that the connections made using Express Route promise speed, reliability, availability and better security compared to connections over the public networks.

A secure network connection can be established with other cloud resources in Microsoft like Microsoft 365, Dynamic 365, and Microsoft Azure, as shown in Figure 4-5.

Figure 4-5. An overview of how Azure Express Route connections between public and on-premises networks, Image Credit: Microsoft Documentation

Express Route has several useful features such as support for different connectivity models between your on-premises network and Azure. As shown in Figure 4-6, these connectivity models can be done using service providers or directly. As for using service providers there are currently three types of models - Cloud exchange co-location, Point-to-point Ethernet connection, and there’s also the Any-to-any (IPVPN) connection. ExpressRoute Direct can also be used for direct connection model to Microsoft Azure.

Figure 4-6. A basic example of the different connectivity models of Azure Express Route

Express Route Direct is a good Azure service ideal if you want to directly connect your network into Microsoft’s network globally at peering locations up to 10-Gps speed of connectivity. You can create ExpressRoute Direct connection through the Azure Portal, Azure CLI, and Azure CLI. Learn more about ExpressRoute Direct circuits, workflows, VLAN tagging, SLA, and pricing.

NAT Requirements for using Express Route

When you configure and manage Express Route, NATs are needed to connect to Microsoft cloud services using Express Route. Connectivity service providers usually offers NAT management as a service. Otherwise, there are some NAT requirements for Microsoft peering and even Azure public peering. There are also some ExpressRoute routing requirements that you need to be aware of.

Azure ExpressRoute has the option to implement private peering for Azure resources like virtual machines and Azure Virtual Desktop RDP Shortpath within the virtual networks.

Azure ExpressRoute Global Reach

Aside from the different connectivity models of Azure Express Route for direct connection and service providers, there is also another type of related networking service called ExpressRoute Global Reach, which is designed for scenarios like connecting an organization’s branch offices across the world.

For example, if your Express Route service provider does not operate in certain locations or countries of your office branch, you may consider using the service provider locally in that location or country. By doing this as an alternative solution, Microsoft will be able to help and provide a solution to connect your organization’s branch offices to your main Express Route service provider in your main office location using its global network and Express Route.

The locations of the ExpressRoute are the starting connection point to the globally distributed network infrastructure of Microsoft. Due to this great capability Express Route users are able to connect around the world using Microsoft’s network.

Azure VPN Gateway

A VPN gateway is a specific type of virtual network gateway that is used to send encrypted traffic between an Azure virtual network and an on-premises location over the public Internet. You can also use a VPN gateway to send encrypted traffic between Azure virtual networks over the Microsoft network. Each virtual network can have only one VPN gateway. However, you can create multiple connections to the same VPN gateway. When you create multiple connections to the same VPN gateway, all VPN tunnels share the available gateway bandwidth.

Different Types of VPN Gateway Connections

In order to create a VPN Gateway connection, we need to know the different configurations for VPN gateway connections. These different types of connections will give you a clear overview and understanding on what is ideal to use based what suits your business requirements.

Site-to-Site VPN (S2S)

The Site-to-Site (S2S) connection runs over IPsec/IKE (IKEv1 or IKEv2) VPN tunnel. IPsec VPN is one one of the common VPN protocols used for establishing VPN connection. The S2S VPN gateway connections can be implement hybrid or cross-premises use case scenarios. This type of gateway also requires a VPN device that must be located on-premises. This VPN device should have a public IP address designated to it.

Point-to-Site VPN

The Point-to-Site (P2S) type of VPN connection is applicable if you want to connect a client computer to a virtual network like Azure VNet. In this the type of VPN connection that needs to be initiated from the client computer which is useful for remote working.

VNet-to-VNet (IPsec/IKE VPN tunnel)

Unlike the P2S VPN connection, the VNet-to-VNet connection is for connecting between VNets, even virtual networks on on-premises. This type of VPN gateway uses secure connection using IPsec/IKE. Configuring for multiple sites connection is also possible in this type of VNet-to-VNet connection.

Site-to-Site and ExpressRoute

The network traffic for the Site-to-Site VPN gateway is securely encrypted through the public Internet. ExpressRoute is also good in creating a private and direct connection from WAN to Azure and other Microsoft services. This means that configuring ExpressRoute together with Site-to-Site VPN have advantages especially if it is set for the same virtual network.

VNet-to-VNet Connections Good to know

The VNets you connect using VNet-to-VNet (IpSec/IKE VPN Tunnel) connections can be in different or same Azure regions, subscriptions, and deployment models.

One of the important steps when configuring the VPN gateway is choosing the appropriate type. A virtual network can only have one VNet gateway and it is important to choose the correct type of gateway. For example, if you are configuring it for VPN, then you would need to use the GatewayType = Vpn and for ExpressRoute would be its keyword value. Check the guide for configuring VPN Gateway settings and some requirements that you need to know.

Azure NAT Gateway for Virtual Networks

Azure Virtual Network NAT (Network Address Translation) helps in simplifying the outbound Internet connectivity for virtual networks (one or several subnets). Once a subnet is associated with a NAT gateway, the NAT gives the benefit of providing a Source Network Address Translation (SNAT). When VNet NAT is configured on a subnet level, the outbound connectivity will use the specified static public IP addresses. Figure 4-7 shows a basic illustration of how Azure VNet NAT gateway works with a virtual machine with public IP.

Figure 4-7. A basic example of an instance-level Public IP with a virtual machine and NAT

The following are some of the benefits of using Azure VNet NAT.

Security and Privacy

Azure VNet NAT Gateway doesn’t need to have public IP addresses; therefore, the connection between virtual networks is fully private and secured. Resources can still connect to external resources outside VNet even without public IP addresses

Scalability

There is a benefit of scalability because all compute resources belong to a subnet. All subnets can communicate and use the same resource in the same virtual network. An example of this is automatic scaling is possible with the use of Public IP Prefix, which assists in identifying and scaling how many outbound IP addresses are required.

Resiliency

One of the good things to know is that NAT does not have any individual dependency on other compute instances which is the result of it being a distributed and fully-managed service. This feature and capability make it very resilient as a software-defined networking service.

Performance

The performance of using NAT Gateway is satisfactory because as mentioned it is a software defined networking service, which means it that when it is running it will not have a negative effect on the network bandwidth.

Azure VNet NAT Gateway provides you with the ability to control who has access to your organization’s resources and what locations they can be accessed. For these reasons, NAT Gateway can be useful whitelisting a group of people working as contractors for a company.

NAT is not applicable and supported to work with basic Public IP addresses and load balancers. If you need to use NAT Gateway in your implementation, you would need to use Standard versions instead or upgrade Azure Public Load Balancers to higher versions.

If you want to try designing a virtual network using NAT Gateway, check Microsoft Docs: Designing virtual networks and a recorded video about it on Azure Friday.

Azure DNS

Azure DNS is a hosting service for DNS domains that provides name resolution by using Microsoft Azure infrastructure. By hosting your domains in Azure, you can manage your DNS records by using the same credentials, APIs, tools, and billing as your other Azure services. You can’t use Azure DNS to buy a domain name. For an annual fee, you can buy a domain name by using App Service domains or a third-party domain name registrar. Your domains then can be hosted in Azure DNS for record management.

The following are some of the known key features of the use of Azure DNS:

Reliability and performance

DNS domains in Azure DNS are hosted on Azure’s global network of DNS name servers. Azure DNS uses anycast networking. Each DNS query is answered by the closest available DNS server to provide fast performance and high availability for your domain.

Security

Azure DNS is secured because is linked to Azure Resource Manager which associated to user identity control service like Azure RBAC and Azure Resource Lock Resource locking is used to prevent other users in your organization from accidentally deleting or modifying critical resources.

Alias records

Azure DNS supports alias record sets. You can use an alias record set to refer to an Azure resource, such as an Azure public IP address, an Azure Traffic Manager profile, or an Azure Content Delivery Network (CDN) endpoint. If the IP address of the underlying resource changes, the alias record set seamlessly updates itself during DNS resolution. The alias record set points to the service instance, and the service instance is associated with an IP address. Also, you can now point your apex or naked domain to a Traffic Manager profile or CDN endpoint using an alias record.

Easy To Use

Azure DNS does not currently support DNSSEC. In most cases, you can reduce the need for DNSSEC by consistently using HTTPS/TLS in your applications. If DNSSEC is a critical requirement for your DNS zones, you can host these zones with third-party DNS hosting providers.

Azure DNS Limitation for DNSSEC

Domain Name System Security Extentions (DNSSEC) are currently not supported in Azure DNS. However, you can have a work-around alternative of using HTTP/TLS in the configuration of your applications. If your DNS zones require DNSSEC, then you choose to host them externally through DNS providers.

By using Azure DNS, you host your own domain websites in Azure, manage your DNS records and integrate it with the resources hosted in Azure. If you have an exist‐ ing domain, there are a number of ways to host it in Azure DNS. Microsoft has different guides for setting up your own custom domain oon a function app, web app, blob storage, or other Azure resources.

Azure Bastion

The Azure Bastion service is a new, fully platform-managed PaaS service that enables secure virtual machine connections. It provides secure and seamless RDP/SSH con‐ nectivity to your virtual machines directly in the Azure portal over TLS. When you connect via Azure Bastion, your virtual machines do not need a public IP address.

Azure Bastion is a service for networking in Azure that will allow you to connect to a virtual machine from a web browser and the Azure portal. It is a fully platform-managed PaaS service provisioned inside your virtual network.

It provides secure and seamless RDP/SSH connectivity to your virtual machines directly over TLS. This means that when you connect to your VMs via Azure Bastion, your VM does not require a public IP address, agent, or client software like we normally do.

You can connect securely via RDP and SSHto all of the VMs in the virtual network where Azure Bastion is provisioned. By doing this, you are securely connecting to your virtual machines and are not exposing RDP/SSH ports to the public Internet.

The following are some of the key benefits and uses of Azure Bastion:

RDP and SSH directly in a web browser

Since Azure Basion uses HTML5-based web client, you can access your virtual machines on any device. This means that you don’t need to download supported RDP or SSH client to connect to a VM. You can connect to your virtual machines securely using RDP and SSH session directly on any web browsers through the Azure portal.

No Public IP Not Required on the Azure VM

Azure Bastion opens the RDP/SSH connection to your Azure virtual machine using private IP on your VM. You don’t need a public IP on your virtual machine.

Save time from managing Network Security Groups (NSGs)

Azure Bastion is a fully managed platform PaaS service from Azure that is hardened internally to provide you secure RDP/SSH connectivity. You don’t need to apply any NSGs to the Azure Bastion subnet. Because Azure Bastion connects to your virtual machines over private IP, you can configure your NSGs to allow RDP/SSH from Azure Bastion only. This removes the hassle of managing NSGs each time you need to securely connect to your virtual machines.

Protection against port scanning

Virtual machines are secured and protected against port scanning by rogue and malicious users located outside your virtual network.

Protect against zero-day exploits

Azure Bastion is a fully platform-managed PaaS service and the Azure platform protects against zero-day exploits by keeping the Azure Bastion protected from exploits by keeping everything updated for you.

Azure Bastion can be considered as a flexible and secured way to connect virtual machines on the go. If you want to learn more about it and how it is used, check out this video guide on how to use Azure Bastion to connect securely to your Azure VMs on Azure Friday.

Services for Application Protection

This section describes networking services in Azure that help protect your network resources. Using any or a combination of these networking services in Azure, you can secure your applications and workloads in the cloud.

Azure Firewall

Azure Firewall is a cloud-native and intelligent network firewall security service that provides the best of breed threat protection for your cloud workloads running in Azure. It’s a fully stateful, firewall as a service with built-in high availability and unrestricted cloud scalability.

The following are the different category types of Azure Firewall which helps in identify which is right for your needs.

Azure Firewall Standard

As shown in Figure 4-8, the Azure Firewall Standard provides Layer 3 to Layer 7 firewall filtering and threat intelligence feeds from Microsoft Cyber Security. These treat feeds can be used to notify any important altert and can even deny network traffic from/to any malicious domains and IP addresses to protech Azure resources against possible hacking or attacks.

Azure Firewall Premium

As shown in Figure 4-9, there is another alternative category called Azure Firewall Premium. This category provides advanced capabilities such as signature-based IDPS that enables detection of attacks by analyzing and detecting for specific patterns. These patterns can include byte sequences in network traffic or any known malicious instruction sequences used by malware. This tier also support 3rd party offerings available from WatchGuard, Sophos, Palo Alto, Check Point, and the like.

Figure 4-8. A basic example of Azure Standard

Figure 4-9. A basic example of Azure Premium

Known issues for Azure Firewall

There are documented and known issues for both the Standard and Premium version of Azure Firewall. These issues and limitations are good to know.

Aside from the different Azure Firewall categories and tiers, you can centrally manage Azure Firewalls across multiple Azure subscriptions using the Azure Firewall Manager. It supports setting up centralized security and route management. This firewall policy can be utilized by applying a common set of firewall rules in your network or application in your Azure tenant. Azure Firewall Manager supports firewalls in environments like VNet and Virtual WANs (Secured Virtual Hub).

Azure Firewall is an excellent choice to secure your Azure resources. I expect that the team behind Azure Firewall will continue to add features to both standard and premium tiers. As mentioned above, I will be leading with Azure Firewall when designing solutions for our customers.

Azure DDoS Protection

Azure DDoS Protection provides counter measures against the most sophisticated DDoS (Distributed Denial of Service) threats that can cause severe and highly impactful loss for its victims. Azure’s DDoS protection service provides advanced DDoS mitigation features for your application and resources.

Furthermore, customers using Azure DDoS Protection have access to support and communicate with DDoS experts using the Azure DDoS Rapid Response during an active attack. Azure DDoS Protection Standard plan is a prerequisite to this support.

Figure 4-10 shows a basic illustration and example of how Azure DDoS protection works in securing an application gateway in a virtual network.

Figure 4-10. A basic illustration of Azure DDos protection for a web application in a virtual network

If you are an engineer with a role involving securing resources in Azure, I recommend learning the basic concepts of Azure security and protecting Azure resources from DDoS attacks.I recommend you to check out Microsoft’s learning path for Introduction to Azure DDoS Protection.

Azure Private Link

Azure Private Link enables you to access Azure PaaS Services (for example, Azure Storage and SQL Database) and Azure hosted customer-owned/partner services over a private endpoint in your virtual network. Traffic between your virtual network and the service travels through the Microsoft backbone network. Exposing your service to the public internet is no longer necessary. You can create your own private link service in your virtual network and deliver it to your customers.

Web Application Firewall

Azure Web Application Firewall (WAF) provides protection to your web applications from common web exploits and vulnerabilities such as SQL injection, and cross site scripting. Figure 4-11 shows an example illustration on how Azure WAF works.

Azure WAF provides out of box protection from OWASP top 10 vulnerabilities via managed rules. Additionally users of WAF, can also configure custom rules, which are customer managed rules to provide additional protection based on source IP range, and request attributes such as headers, cookies, form data fields or query string parameters.

Figure 4-11. A basic example of Azure Web Application Firewall

Users or Azure administrators can choose to deploy Azure WAF with Application Gateway which provides regional protection to entities in public and private address space. It is also possible to deploy Azure WAF with Azure Front Door which provides protection at the network edge to public endpoints.

Network Security Group (NSG)

Network Security Groups (NSGs) are built-in tools for network control that allow us to control incoming and outgoing traffic on a network interface or at the subnet level. They contain sets of rules that allow or deny specific traffic to specific resources or subnets in Azure. An NSG can be associated with either a subnet (by applying security rules to all resources associated with the subnet) or a Network Interface Card (NIC), which is done by applying security rules to the Virtual Machine (VM) associated with the NIC.

Figure 4-12 illustrates how NSG is being used at different levels to protect several virtual machines in different subnets in a virtual network.

Figure 4-12. An example implementation overview of a virtual network protected with NSG

Azure Load Balancer

Azure Load Balancer is a load balancing networking service that provides the capability of distributing traffic across multiple virtual machines (Azure VMs) or a group of resources in a single Azure region. This resource is known for its best performance with ultra-low latency. The algorithm for Azure Load Balancer uses a tuple-based hashing distribution, which means that the tuple hashes itself on the bases of its elements.

Figure 4-13 shows us an example of a 5-tuple hash (source IP, source port, destination IP, destination port, and protocol type) that are to be mapped to the traffic to any available servers.

Figure 4-13. An illustrative example of Azure Load Balancer with 5-tuple hash distribution

The hash-based and source IP affinity are both supported in the Azure Load Balancer service. This means that you have the option to set up the configuration of your preferred distribution mode for traffic distribution.

Key Uses of Azure Load Balancer

Azure Load Balancer, as shown in Figure 4-14 is a powerful resource to use of creating applications that are highly available and scalable for inbound and outbound connections.

The following are some of the uses and benefits of using the standard Azure Load Balancer:

Improve the distribution of Azure resources with better availability

Configure connectivity and load balancing of Azure VMs for incoming and outgoing traffic

Monitoring of Azure resources that are being load-balanced and distributed by the Azure Load Balancer

Require port forwarding, load-balance with IPv6, multiple ports or IP addresses

Ability to migrate load balancing resources across different Azure regions

Figure 4-14. An example of a hash base type of load balancing in Azure.

The load balancer operates in Open Systems Interconnection (OSI) model at layer 4 which distributes traffic based on protocol, source IP address and port numbers and destination IP address and port. The main purpose of load balancer is to distribute the inbound flows from both backend and frontend pool instances.

Azure Load Balancer has some important components, as described in Table 4-15, a few key components which can be configured in your Azure subscription on the Azure Portal. You also have the option to configure it using other Azure tools like Azure Powershell, Azure CLI, or ARM templates.

Table 4-1. Azure Load Balancer Components

Component
Description

Frontend IP Configuration

The IP address (Public or Private) you set on a Load Balancer will be your client’s access point. The type of load balancer depends on what kind of IP address you have. For example, the Private IP address for the internal load balancer while the public IP is for the public load balancer

Backend pool

It is generally recommended to add more instances to the backend pool in order to cost-effectively scale and handle high demands of incoming traffic. To optimize the operations, it is worth considering designing for the least number of individual backend pool resources

Load Balancer rules

A load balancer rule configured with protocol - all and port - 0 is known as a High Availability (HA) port rule. This rule enables a single rule to load-balance all TCP and UDP flows that arrive on all ports of an internal Standard Load Balancer. The HA ports load-balancing rules help you with critical scenarios, such as high availability and scale for virtual network appliances (NVAs) inside virtual networks

Health probes

A health probe is used to determine the health status of the instances in the backend pool. During load balancer creation, configure a health probe for the load balancer to use. This health probe will determine if an instance is healthy and can receive traffic

Inbound NAT rules

An inbound NAT rule forwards incoming traffic sent to frontend IP address and port combination. The traffic is sent to a specific virtual machine or instance in the backend pool. Port forwarding is done by the same hash-based distribution as load balancing

Outbound rules

An outbound rule configures outbound Network Address Translation (NAT) for all virtual machines or instances identified by the backend pool. This rule enables instances in the backend to communicate (outbound) to the internet or other endpoints.

High Availability Ports

The HA ports load-balancing rules help you with critical scenarios, such as high availability and scale for network virtual appliances (NVAs) inside virtual networks. The feature can help when a large number of ports must be load-balanced

Azure Load Balancer with Several Frontends

If you need a need to set up load balancers in more than one frontend, you can also configure it for multiple frontends, check out Microsoft’s guide for Multiple frontends for Azure Load Balancer

Load balancing is a term commonly used for workload distribution across several computing resources. This process helps optimize the uses of resources, fix issues with response times, increase throughput, and prevent single resource overloading. In Azure, there are other known services for load balancing such as Traffic Manager, Azure Front Door, and Application Gateway.

To learn and investigate more on which load balancing network service is suitable for specific scenarios, check out Reviewing Network options and identitifying requirements for workload networking.

Figure 4-15. An example of Microsoft’s decision tree guide for the different load balancing options. Image Credit: Microsoft Documentation

Each application can have different implementations and can consists of multiple workloads. Each workload needs to be evaluated separately. There are several load balancing options in Azure, the load balancing decision tree guide provided by Microsoft is shown in Figure 4-14. You can also read the details of this flowchart in the Microsoft’s guide for Load Balancing Options.

Azure Networking Services for Application Delivery

Application delivery is another group category for Azure networking services. The following networking services in Azure, like Azure Front Door, Azure CDN, Azure Balancer, etc., will help speed up delivery and user experience on your cloud-hosted applications globally.

Azure Front Door

Azure Front Door is a global, scalable entry-point that uses the Microsoft global edge network to create fast, secure, and widely scalable web applications. With Front Door, you can transform your global consumer and enterprise applications into robust, high-performing personalized modern applications with content that reach a global audience through Azure.

By making use of the Azure Front Door service, the content on your applications will be delivered quickly to your application users wherever they are located. This service uses Microsoft’s global edge network that interconnected to many POPs distributed around the world.

With Front Door you can develop, operate, and scale out your dynamic web applications and even static content. It also allows you to configure, manage, and monitor the global routing for your web traffic by optimizing for top-tier end-user performance and reliability through quick global failover.

The following are the additional key benefits and uses of Azure Front Door:

Cookie-based session affinity which is useful for keeping a user’s session active on the same device or server

SSL offloading and certificate management

Custom domains if you want to configure your own domains

Integration with Web Application Firewall (WAP) for security

HTTP/HTTPS redirection

URL re-writes and custom redirection

Smart monitoring for resources in the backend that will help track and debug issues

Multiple websites hosting and support for wildcard domains

End-to-end IPv6 connectivity and HTTP/2 protocol support

There are different tiers to choose from when using this service, the Azure Front Door Standard and Azure Front Door Premium. If you want to find out the details of the different tiers, check out Microsoft’s guide for Azure Front Door Tiers.

Azure Application Gateway

Azure Application Gateway is a load balancer for web traffic. It enables you to manage and control the traffic to your web applications. It is an Application Delivery Controller (ADC) as a service that supports several layer 7 load-balancing capabilities for your applications.

Microsoft’s Azure Application Gateway is ranked is top #3 in Application Delivery Controllers solutions based on Peerspot.com ranking survey

The following are some of the features of Application Gateway:

Support for autoscaling (Scale up and Scale down) based on the ongoing traffic load of your application

SSL/TLS termination at the gateway to secure unencrypted traffic to servers

Zone redundancy and support for multiple Availability Zones to ensure high availability and fault tolerance

URL-based routing, multiple site hosting and redirection

Protection and security using Web Application Firewall (WAF)

Use as Ingress Controller for Azure Kubernetes Service (AKS) cluster

Integration with Azure Monitor for monitoring, logging and insights

Azure has several fully-managed load-balancing solutions that suit different use case scenarios or needs. Azure Load Balancer is an alternative if you want to do load balancing at the network layer level. There’s also Azure Front Door for optimizing global routing for your web traffic and Azure Application Gateway if you need server load balancing at the application layer.

Azure Traffic Manager

The Azure Traffic Manager is a DNS-based traffic load balancer. This networking service allows you to effectively distribute traffic to your public-facing applications across the global Azure regions. Traffic Manager also provides your public endpoints with high availability and quick responsiveness.

Let’s take a look at some of the key features of Azure Traffic Manager:

Integrate and associate hybrid applications

Traffic Manager supports external, non-Azure endpoints enabling it to be used with hybrid cloud and on-premises deployments, including the “burst-to-cloud,” “migrate-to-cloud,” and “failover-to-cloud” scenarios.

Improve availability, maintainability and performance

Azure Traffic Manager promises to deliver high availability for critical applications. It is done by monitoring endpoints and automatic failover if an endpoint goes down. You can have planned maintenance done on your applications without downtime. Traffic Manager can direct traffic to alternative endpoints while the maintenance is in progress. It can help improve your application’s performance by directing web traffic to the endpoint with the lowest latency.

Manage complex traffic distribution and advanced deployments

By using Mested Traffic Manager profiles, you can use multiple traffic-routing methods to create flexible rules to scale to the demands of more complex and larger deployments.

Different routing methods for Traffic Manager

Priority, Weighted, Geographic, Performance, Multivalue, Subnet are the traffic-routing methods supported by Azure Traffic Manager. These routing methods are used to check how the network traffic are routed to different service endpoints.

Figure 4-16. An example of Azure Traffic Manager being used for a web application to scale and load balance traffic on hybrid cloud environment

The Figure 4-15 shows an example of how Azure Traffic Manager can be used to direct web application’s client request to the a specific endpoint based on the traffic-routing method being configured.

Important note about Traffic Manager vs. Front Door

Both the Azure Traffic Manager and Azure Front Door have similarities but also differences. For example, when it comes to routing options, Traffic Manager uses On-premise routing at a DNS-layer; while Azure Front Door works with HTTP-requests which has an independent scalability. As for protocols, Traffic Manager works with any protocol like UDP, TCP, HTTP, and more. In contrary, Front Door uses HTTP-acceleration which mean that its traffic is being on proxy at the edge network of Microsoft.

If you have several instances in different Azure regions and if one of these instances fails during the health check, it is expected that the traffic for your application is directed to the Azure region that is healthy. However, a performance problem can occur in case of latency of the traffic to a region that is located far away.

Therefore, there are some performance considerations to think about when using and managing Azure Traffic Manager. There are several tools that you can use to measure your DNS latency and performance. Discussing the technical details of these traffic monitoring tools is beyond the scope of this book. If you want to learn more about it, check out Microsoft’s recommendation for Measuring Traffic Manager Performance.

Azure CDN

Azure Content Delivery Network (CDN) is a global CDN solution for delivering high-bandwidth content by caching these content in different locations. You can choose to use Azure CDN for applications hosted in Azure or any other servers or location.

The Azure CDN enables you to cache static objects which can be coming from your web applications, Azure Blob storage, or a web server that is publicly avaialable. It uses the closest point of presence (POP) server, as shown in Figure 4-16, which helps accelerate dynamic content.

Figure 4-17. Example of POP (Point of Present) with Azure CDN.

The following are some of the features of Azure CDN:

Improved and better performance and user experience for end-users globally

Gain the ability to scale to handle instantaneous high load and demand

Azure storage blobs can be used to cache content to make it publicly available

CDN caching of web content, images, scripts, and other web assets

Allows you to cache content based on specific query strings

Access the cached content from a custom domain through CDN HTTP Endpoint mapping

Azure CDN have different tiers such as Azure CDN Standard from Microsoft, Azure CDN Standard from Akamai, Azure CDN Standard and Premium from Verizon. Prior to setting up your Azure CDN account, you should understand the different tiers and the comparison of the different features they offer.

Azure Networking Services for Network Monitoring

This section describes networking services in Azure that help monitor your network resources such as the Azure Network Watcher, Azure Monitor Network Insights, Azure Monitor, ExpressRoute Monitor, etc. We will take a look at the overview of each of these services and how to use them to monitor your network.

Azure Network Watcher

Azure Network Watcher is a composed of different tools used for monitoring, diagnosing, logging, and metrics management for an Azure Virtual Network. This networking service is commonly used to track and monitor network health statuses of IaaS services such as virtual machines, application gateways, load balancers, virtual networks, etc.

The following are some of the benefits of using Azure Network Watcher:

Monitor communication between virtual machines and network endpoints at regular intervals with alerts and notifications.

Get a visualization of the overview and the relationships of Azure resources in a virtual network

Detect and troubleshoot any network traffic filtering problems between your VMs

Diagnose problems of network routing and outbound connections for virtual machines

Virtual machines packet capturing that helps detect possible network anomalies

To highlight, Azure Network Watcher is useful when you are troubleshooting issues related to or detecting network traffic anomalies in Azure IaaS resources. For example, if you want to troubleshoot a problem with a VPN connection between two virtual machines, you can use Azure Network Watcher to monitor the traffic between the two virtual machines.

Finally, there is one limitation of using Azure Network Watcher that is worth knowing. It is not going to work with web analytics and monitoring PaaS services because this service was not designed to be used with these two services.

Azure Monitor Network Insights

Azure Monitor Network Insights provides an overall view of health and metrics for all the network resources that you have deployed and hosted in Azure. This network insights service is part of Azure Monitor that allows you to track networking metrics easily without any advanced configuration.

It is structured around these key components of monitoring:

Network health and metrics for the visualization of all your networking resources with the option to search, filter, setup alerts and dependency view

Connectivity helps you visualize all the configured tests done via Connection Monitor

Traffic gives you a view of all flow logs for Network Security Groups (NSGs) and also the Traffic Analytics for the selected set of subscriptions, grouped by location.

Diagnostic Toolkit Diagnostic can be used in troubleshooting the network issues such as IP flow verification, packet capture, etc.

If you are using Azure Monitor for monitoring and gathering metrics for your Azure resources, then network insights is good tool to use especially when you have multiple networking resources hosed in Azure.

Azure Space - Networking beyond the Clouds

Azure Space is one of the most-advanced innovations of Microsoft. Microsoft wanted to create networking and connectivity beyond the clouds, to the space. It was created and serves as an eco-system and platform for the space community.

As of the date of writing, there are three services created for Azure Space: Azure Orbital, Azure Modular Data Center, and Azure Orbital Emulator.

Figure 4-18. An illustration of how the global satellites and networks are connected using Azure Orbital.

Azure Orbital

Azure Orbital is a fully-managed Ground Station As-a-Service (GSaaS) under the category of Azure Space. As of this day of writing, this service is currently on the Preview version. It helps users to communicate, downlink, schedule service and control their satellites, and scale operations from Azure. With the use of Azure Orbital, integrated data processing is easy and managed on the Azure platform. Data being processed are transferred securely through the user’s virtual network, which can be stored on Azure Storage or any Azure service. The main uses of Azure Orbital is for global communications and earth observation Figure 4-18 shows us an illustration of how Azure Orbital (GSaaS) is connected to Azure Data centers using Azure WAN and connects to the Global Communications and Orbiting satellites up in space using an orbital downlink.

Azure Orbital Emulator

Azure Orbital Emulator is the tool that enables satellite developers to use AI algorithms and evaluate before launching. Complex satellite networking can be simulated by this emulator. The Orbital Emulator assists in getting simulations and perspective on how the application functions when it is in orbit. This tool enables the satellite developers to test their applications in the cloud (Azure) first before they deploy them to space.

Azure Modular Datacenter

The Azure Modular Datacenter(MDC) was designed and created to help users who need cloud computing solutions in hybrid, challenging environments and remote locations. As a user of MDC, you have the capability to deploy your own self-contained transportable data center unit near the location you need it to be. In simple terms, Azure provides a portable data center unit (MDC units) that is fully equipped with full network connectivity and secure network connectivity using satellite communications.

Finally, if your organization is interested in learning more about the Azure Orbital and cloud services for the spaces, I recommend checking out about their Azure Space Partner Community program.

Learn By Doing (Try it out!)

Aside from the supplementary learning hands-on lab repository for the topics in the this chapter, the following quick-start tutorials are most recommended as they are updated based on Microsoft’s technical updates for the service. This section is recommended and optional.

Quickstart: Create a virtual network using the Azure portal

Create a Site-to-Site Connection in Azure Portal

Quickstart: Create a public load balancer to load balance VMs using the Azure Portal

Tutorial: Create a site-to-site VPN connection in the Azure Portal

Create and provision an ExpressRoute circuit

Summary

In this chapter, we went through the basic concepts, descriptions, features and even limitations of the most common networking services and tools in Azure. We learned that networking services are categorized by its different purposes e.g., connectivity, application delivery, security and monitoring. These networking services grouped by categories give us an overview and idea on which right networking Azure service to use along with compute resources in your cloud application development.

Azure has several load balancing solutions Front Door, Traffic Manager, Application Gateway, or a combination of any. We also learned that Azure networking goes beyond the clouds of cloud computing to the space through Azure Space services like Azure Orbital, Azure Orbital Emulator and Azure Modular Datacenter that are offered as Ground Station As-a-Service (GSaaS). GSaaS allows space explorers and researchers to communicate globally through space stations for research and innovation purposes.

In the next chapter, we will learn more about the different storage and database solutions in Azure.

Check Your Knowledge

What is the basic and fundamental block of a private network in Azure that enables your Azure resources to communicate with each other?

How can you protect and secure your applications in Azure using the networking resources available? (List down a few examples)

You need to securely connect your on-premises network to Azure in a private connection; which Azure networking services would you use?

What networking service would you use if you want to filter inbound and outbound traffic of Azure resources and securing your VNets?

What Azure networking service would you use if you want to features like Dynamic site acceleration, file compression, geo-filtering, and set caching rules for web content?

To find the answers to the Check Your Knowledge, turn to the back of the book in the Appendix

Recommended Learning Resources

Azure Networking Fundamentals

Explore Azure Networking Services

Microsoft Learn: Architect network infrastructure in Azure

Networking Services YouTube Tutorial by Adam Marczak

Microsoft Azure Master Class - Networking by John Savill

Azure Networking Architecture Documentation

Working Remotely with Azure Networking Services

Azure CDN Documentation

Azure Orbital Documentation

MC2MC Community: Zero to Hero with Azure Virtual WAN by Derek Smith

Microsoft Azure Networking Book by Avinash Valiramani

Microsoft Documentation: Azure Orbital

E-Book: Kubernetes on Azure

Chapter 5. Microsoft Azure Cloud Storage and Databases

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 5th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at jleonard@oreilly.com.

At the heart of any application is the data, and whether that data is an image, an arbitrary fact about lumpia, or something in between, it needs to be stored somewhere! Luckily, there are plenty of data storage and database options that suit specific needs. Which do you use? That will depend on some key factors, like how you intend to interact with your data, how much of it you need to store, what kind of data you are storing, and how your application is structured. In the end, choosing the right database and data storage options can mean the difference between a resilient, scalable, and delightful application... and one that is not, so choose wisely!

Adrienne Braganza Tacke, Senior Developer Advocate and Software Engineering Leader at Cisco

In the previous chapter, we learned about the importance of networking in the cloud infrastructure of Microsoft and the different Azure networking services for different important purposes. Implementing databases services like relational databases and Microsoft SQL Services has been a great part of the software development with Microsoft technology stack. ¹

In this chapter, we will learn about the benefits of cloud storage in Azure and the different cloud database services for both the different types of data. Cloud storage services are important in our personal daily practical data management, software development and cloud engineering today. Specially for handling our data, managing it and protecting our data operations securely. By the end of this chapter, you will gain knowledge about the different data storage and database solutions that you can use for your modern web applications and data storage hosted in Microsoft Azure.

The Importance of Data Storage and Databases in the Cloud

In the data-driven generation today, organizations and businesses consider data to be the most available asset. Having a good set of strategies for storing data for our enterprise applications and systems is critical and vital for the success of businesses and organ‐ izations in this modern era.

Cloud computing has been helpful to many organizations moving forward to digitalization, and also cloud solutions have helped bioscience research, innovation, and collaboration.² Cloud computing aids in collaborative research by providing secure access to important digital tools and data management systems with powerful, reliable and scalable computing resources.

The volume of enterprise data grows exponentially, and the value and importance of data are crucial for businesses today. The data we store of any kind, e.g., analytical data, financial data, client data, vendor data, etc., continues to expand when our businesses grow. Therefore, it is important to have a sustainable data storage solution that can handle the data we store.

The traditional way of storing data will soon fade away, and storing data in the cloud will be the new way of storing data. Storing our data in the cloud provides the benefits of highly scalable, secure storage for business applications, systems, and workloads that will also help in cost-effectiveness in the long run.

Data professionals, e.g., database administrators, data engineers, data scientists, and data analysts, play important and vital roles in your data-driven organizations, society, and cloud solutions today. These career roles are essential to the business, and they are involved with the engineering and management of the enterprise data that we store in your digital storage or databases through our systems and applications.

If you are a data professional working with data in your job, then reading this book this chapter will help you accelerate your skills in data engineering and cloud development by learning more about the different solutions for data storage and databases in Microsoft Azure.

Data Storage Management in the Cloud

Data storage management is crucial in any organization today. Users of your systems or applications need to be able to access and manage their data whenever and wherever they are. Therefore, storing data in any cloud platform, like Azure, has been an effective solution and option for data storage management. Another factor that is important when it comes to data storage management is the policy for data retention.

Data retention policy helps organizations define which data they need to retain for security, compliance, and operational reasons. These retention policies are important to have, especially when the business data continues to increase and grow exponentially. Storage management for data, e.g., structured or unstructured, will assist in resource provisioning, configuration, cost-management, and evaluation.

Using Microsoft Azure as a cloud storage provider, you can easily manage your data storage and its retention policy for your Azure resources. For example, if you are using Azure Storage, you can set up time-based retention policies for immutable blob data.

Why Digital Storage in the Cloud is Useful

There are many good reasons why cloud storage is beneficial to us and businesses ³. Nowadays, storing data digitally and cloud storage has been the most efficient and convenient for us. Cloud storage allows users to minimize the use of using a physical local hard disk or storage that has the risk of getting lost. Storing data digitally ena‐ bles global and remote access with dynamic options to upgrade the storage capacity on-demand when we need it.

Enterprises and businesses can save money ⁴ by using cloud storage services. They don’t need to spend money on the expenses of storing their data in their own servers or data centers. Cloud computing providers offer these storage services with features of security, reliability, and scalability at a global scale.

Some of the other reasons why cloud storage is beneficial are the synchronization of data securely stored on the cloud, which makes it easier for users to share and access important files or content across any device from anywhere. Aside from the flexibility of upgrading your storage anytime, there is also a great advantage of better storage management with better data backup management and disaster recovery benefits that the cloud provider offers. So in case of an emergency of data loss, you can worry-free retrieve and restore your backup data from the cloud.

Big Data, Structured Databases, and Non-Structured Databases

As mentioned earlier, the data we collect and store can expand enormously as our demands and businesses grow. The term Big Data is a commonly used word to describe large volumes of data that can be difficult to manage. As cloud computing continues to evolve, so is big data, which helps in many ways. For example, data can be used to analyze and improve a business’ customer service strategy. ⁵

Also, these days, the data that we generate and are collected through our applications in the cloud are enormous and getting huge, which means we need to find ways to handle the complexity and heterogeneity of these varieties of data. Using our traditional relational databases has its challenges in handling such massive unstructured data that we need to handle and manage.

In order to understand the importance of big data, we need to understand the different types of data that we can store in the cloud. The different known data types are structured data and unstructured data.

Structured Data

Structured data (SQL) is highly-organized and formatted so that it’s easily searchable in relational databases. Unstructured data has no predefined format or organization, making it much more difficult to collect, process, and analyze. Examples of relational database systems are Microsoft SQL Server, Oracle, MySQL, Azure SQL Database, Postgres, Azure SQL Database, and many more. Structured databases like SQL have their limitations in handling other types of data.

Non-structured Data

Non-structured (NoSQL) data is a type of data that does not have structure. It means it is without a predefined format. Usually, these types of data are considered qualitative types of data like video files, text files, audio files, mobile activ‐ ity history logs, posts on social media, images from a satellite, etc. It is a type of data that is hard to deconstruct because it has a predefined structure. This means that unstructured data cannot be organized and structured the way we usually do it on typic relational databases like SQL. The common types of databases that are suitable for storing and managing non-structured data are the non-relational or NoSQL databases. In the world of open-source, for example, there are common types of NoSQL databases like key-value stores, document databases, graph data‐ bases, and column databases. A data lake in Azure can also be used to analyze and manage unstructured data.

Now that we have learned the different types of data that we can collect and store digitally, we will be able to easily identify and make good better decisions on which is the right Azure cloud storage or databases are suitable for the data we collect in our applications or systems, regardless if they are stored on an existing on-premise databases or already hosted on the cloud.

Figure 5-1 shows an illustration about the difference of structured databases and non-structured SQL databases.

Figure 5-1. Structured vs. Non-Structured Databases

Making sense of the difference - Structured vs. Unstructured Data

Structured data is quantitative type of data that is highly-organized and structured for easy searchable in traditional relational databases that we are familiar with. On the other hand, Unstructured data is qualitative which is considered unstructured and no specific predefined format nor structure. Most of the time, structured data are easier to collect, process and analyze compared to unstructured data.

Being able to understand whether the data type of data e.g., structured or non-structured is important to know not just from a business and knowledge perspective but also in finding ways and good judgment on what is the ideal type of cloud storage service or database solution that you need when storing and managing these data to a cloud platform such as Microsoft Azure.

Azure Storage and Database Services in the Cloud

Microsoft has a great variety of cloud storage solutions for different types of data storage use case scenarios. Our applications and systems need storage and databases to store, manage, analyze, and visualize enterprise and application data. There are solutions available for storage, databases, visualization and analytics of data in Azure.

Azure Storage

The Azure Storage platform is a cloud-based fully-managed data storage solution in Microsoft Azure that allows users to store and manage their data in the cloud. Azure Storage provides durable, secure, scalable, and highly-available storage via HTTP, HTTPS, or REST API for any type of data of any size in your organization.

Azure storage has several benefits to organizations, including its users of any roles. The cloud storage platform in Azure provides secure and durable data storage management in the cloud by ensuring that the data stored in the cloud is safe, secured, and available in case of unexpected system failure or emergency. Users can replicate their data across different geographic locations and data centers for backup and disaster recovery.

Another great feature of Azure Storage is that it is created and built to be scalable, which means that it is able to handle the performance, capacity, and storage demands of your organization. It is designed to be flexible and can be scaled up and down to meet your business needs. Using Azure Portal and the free cloud storage tool Azure Storage Explorer you can easily administrate your data in Azure Storage from the cloud directly from your computer on any operating system.

IT professionals, data engineers, and developers have great productive tools to use with Azure storage, such as Azure storage client libraries to build APIs and web services with Azure Storage platform using the different support languages supported like C#, Java, Python, C++, Typescript, and many more. Cloud engineers and Azure developers can also build scripts, configuration, and automation tasks using the widely used tools in Azure like the Azure Powershell and Azure CLI.

The different types of Azure Storage Data Services and their specific use cases and purposes will be discussed in detail in the later sections of this chapter.

Database Services in Azure

Aside from Azure Storage there are other fully-managed solutions in handling the data we collect and store from and to our applications. Azure provides different types of cloud database services in Azure that are known to be powerful, reliable and scalable. There are different options for both relational, non-relational (NoSQL) and in-memory database solutions available for modern cloud application development use case scenarios for your organization.

Services for Azure Storage

We have learned that Azure has different types of services for both cloud storage and databases. In this section of the book, we will learn about the different types of services are available for cloud storage depending on the type of data your organization needs to store and manage. Table 5-1 shows some of the common Azure storage services that you can use for different use cases, purposes and of course the type and the size of data you are trying to collect and store.

Table 5-1. Different Types of Azure Storage Data Services

Name
Description and Purpose

Azure Blob Storage

If you need a massive object store that should be scalable then storing blobs e.g text and binary data would be ideal using Azure Blobs. It is ideal for example if you want to store and manage your BLOB files on the cloud on any accessible web browsers on any platform for distributed access for external users or applications. Big data analytics through Data Lake Storage as well as backup and recovery solutions are also supported in this service.

Azure Files

If you need a file storage service that is ideal for storing and managing files on any accessible web browsers on any platform for distributed access for external users or applications. Azure Files is a file share in Azure that is fully managed and can be accessed using some of the standard protocols like Network File System (NFS) and Server Message Block (SMB) protocols.

Azure Queues

If you need to securely store large numbers of messages between applications that are reliable. Azure Queues enable you to store messages in queues to be processed between components of your applications. These queue messages can be accessed securely using- REST, HTTP, or HTTPS protocols.

Azure Tables

If you need to store non-structured or schemaless store of your structured data that can be accessed authentically internally and externally the cloud infrastructure in Azure then Azure Tables would be ideal to use.

Azure Managed Disks

If you want to handle and manage large block-level storage volumes or if your workloads need a support for virtual disk features and storage for infrastructure as a service (IaaS) or virtual machines deployment.

Azure Blob Storage

Azure Blob Storage is a cloud-based object storage service that allows you to store and manage your BLOB data in the cloud. This service is able to handle massive text and binary data, especially unstructured types.

Some of the common uses of Azure Blob Storage are managing images, videos, or files on any web browser and device, if you need to store files for external distribution or access either for business purposes or personal use.

If you want to stream BLOB files like videos or audio files for your video streaming services or podcast, you can also use the Azure Blob Storage service to access these data from your application or directly from your browser. Another important use is also for performing a backup and archive of data which is useful and important in data management.

One of the most important concepts you need to know when you are setting up your Azure Blob Storage is its different components of it. Blob Storage has three types of resources: the Azure storage account, the container in this storage account, and then the data object or BLOB items that you store in this container, as shown in Figure 5-2.

Let’s check what are the different components of Azure Blob Storage:

Storage Account

You need a storage account primarily because it is a unique namespace for your BLOB data. Azure Storage account is important because it actually provides a unique URL address or link to every object data you store in your storage. Usually when you setup or create your storage account, you would need to decide and choose a unique account name. The Azure Storage blob endpoint and account storage name becomes the base link address for the data objects in your storage account. If your storage account is named learningmicrosoftazurestorage, then the default endpoint link for Blob storage is learningmicrosoftazurestorage.blob.core.windows.net.

Containers

In an Azure Storage Account, there is usually a container that is associated and saved under its level. You can have more than one or unlimited number of containers in a storage account. Also, in each of your container, there is no limitations to number of data objects or BLOBs that you can store. Containers are where you organize your set of blobs just like how we organize our photos or videos traditionallly in a standard file system. Basically, containers are like our traditional digital folders on our computers. It has a resemblance of a file system directory.

Blobs

There are different types of blobs that Azure Storage supports: Block blobs, Append Blobs and Page Blobs. The Block blobs are primary used to store text and binary data.that that can be individually handled. The Block blobs can handle in storing blob up to the size of 190.7 TiB. The second type of blob is called Append blobs, which are composed of block blobs specially used for append operations. These types of blobs can be used for scenarios like VM data logging. The third type are the Page blobs which are useful in storing random access files up to 8TB for example it can store Azure Virtual Machines virtual hard drive (VHD).

The data objects you store in Azure Blob storage is accessible anywhere worldwide throuh secured and authenticated protocols like HTTPS. Your development team or your organization can implement application solutions with Azure Blob Storage by making use of the development tools and technologies supported. Such technologies including using client libraries for different programming languages like .NET, Java, Python, Node.js, Ruby, PHP, etc. Data can be accessed securely using REST API, Azure CLI, and Azure Powershell.

Naming conventions to be mindful of when using Azure Blob Storage

The container where you store your BLOB objects have its properties and metadata too. There are some naming conventions and best practices you need to consider when working with Azure Blob Storage. To find our more about it, check our Microsoft’s Documentation Naming and Referencing Containers, Blobs, and Metadata

Azure Files

Managed file shares for cloud or on-premises deployments. Azure Files offers fully managed file shares in the cloud that are accessible via the industry standard Server Message Block (SMB) protocol or Network File System (NFS) protocol.

Azure Files file shares can be mounted concurrently by cloud or on-premises deployments. SMB Azure file shares are accessible from Windows, Linux, and macOS clients. The NFS Azure Files shares are accessible from Linux or macOS clients. Additionally, SMB Azure file shares can be cached on Windows Servers with Azure File Sync for fast access near where the data is being used.

Table 5-2 shows some of the common use cases and benefits of using Azure Files.

Table 5-2. Common and Practical Use Cases of Azure Files

Uses and Benefits
Description

Replacement of on-premises file servers

For easy sharing and global access Azure Files can be used as an alternative to Network-attached Storage(NAS) or tradional storage on-premises

Flexible option to Migrate Applications

It is easy to “lift and shift” applications to the cloud that expect a file share to store file application or user data. Azure Files supports both the “classic” lift and shift and the “hybrid” lift and shift scenarios

Simplify cloud development

Run stateful and long-running background jobs or tasks with autoscaling capabilities based on CPU or memory usage

Persistent Volume for Containerization

Containers deliver “build once, run anywhere” capabilities that enable developers to accelerate innovation. For the containers that access raw data at every start, a shared file system is required to allow these containers to access the file system no matter which instance they run on.

Azure Queue Storage

Azure Queues is a storae service that is useful for storing large numbers of messages in the cloud that you can access and manage anywhere securely. A queue message can be up to 64 KB in size and can contain millions of messages depending on the storage capacity allowed on your account.

There are some components in Azure Queues: Storage account, URL format, Queue and Message.

URL format

The URL format of Azure Queue Storage are endpoint or URL in this URL format: storageaccountname.queue.core.windows.net which means that if you want to access an image name called my-image from a queue storage, you may access it using the following address: storageaccountname.queue.core.windows.net/my-image

Storage account

An Azure storage account stores the data objects for your queue messages in the storage account that you created. The storage account is a unique namespace that allows you to access your storage anywhere over secured protocols over the Internet like HTTP or HTTPS. Azure storage account is known for its durability, security, scalability, and high availability.

Queue

Usually a set of message can be contained in queue. When it comes to setting up names for your queue storages there are naming conventions to be mindful of. For example, the queue name is required to be in all lowercase

Message

A message in an Azure Queue Storage can be in any format (up to 64 KB) and the allowed maximum time-to-live(TTL) before version the the date 2017-07-29 is seven days and for later versions after that is the maximum TTL can be any positive number, or -1 indicating that the message doesn’t expire. When you miss setting this TTL parameter, usually the default would automatically be seven days.

Azure Table Storage

If your organization needs to collect, store and manage large amounts of structured data, then Azure Table Storage would be ideal for handling it. Azure Table Storage is a NoSQL data storage that is capable of internal and external authenticated calls from and to Azure. Azure table storage is ideal for storing structured data or relational data.

Some of the usual cases you can use Azure Table Storage includes:

Collection and storage of terabytes(TBs) of structured data for web applications

Data access using the OData protocol and LINQ queries for WCF Data Services in .NET Libraries

If you want to store easy to denormalized and non-complex datasets

Quick data queries using clustered indexing

Azure Table Storage has several components: Storage account, URL format, Table and Properties. Let us go through the definition of each of the components.

Storage Account

All access to Azure Storage is done through a storage account. All access to Azure Cosmos DB is done through a Table API account.

URL format

Azure Table Storage accounts use this format for its URL: http://mystorageaccount.table.core.windows.net. If you are using the Azure Cosmos DB Table API account with this service, the URL format is http://mystorageaccount.table.cosmosdb.azure.com/. Using OData protocol you can also directly access Azure table storage using these URL formats.

Table

A collection of entities is called a table which actually doesn’t implement and enforce a schema. A table can have entities with different sets of properties.

Properties

A property is a name-value pair. Each entity can include up to 252 properties to store data. Each entity also has three system properties that specify a partition key, a row key, and a timestamp. Entities with the same partition key can be queried more quickly, and inserted/updated in atomic operations. An entity’s row key is its unique identifier within a partition.

Azure Table Storage will automatically scale as demand increases especially if you need to use it for storing and querying large sets non-relational or structured data.

Work with the data in Azure Storage using Azure Storage Explorer

To manage your data or files easily on your Azure Storage Account. You can make use of the free stand-alone app Azure Storage Explorer on any platform like Windows, macOS, and Linux.

Azure Managed Disks

Azure Managed Disks are recommended solution if you have block-level storage volumes for IaaS Azure VMsand Azure VMware solutions. This cloud storage solution is virtualized disks on the cloud. You can literally consider it as a virtualized version of your on-premises physical storage. These virtual managed disks in Azure are fully managed, which means that when you set it up, you would need to choose some of the properties like the size, the location, the type of the disk, e.g., ultra disks, standard SSDs, premium solid-state drives (SSD), etc. and the operating system you want to install on the disk.

Table 5-3 describes good uses of Azure Managed Disks.

Table 5-3. Known Benefits of Azure Managed Disks

Uses and Benefits
Description

Availability, High Durability and Scalability

Managed disks are built to be really durable by assuring and securing several data replications, at least three replications which gives high durability and promises avalability of 99.999% in cases of data loss or failures

Availability Sets and Availability Zones Integration

Availability sets is supported and can be integrated with Azure Managed Disks to solve the problem of single point of failure through isolation. Azure Availability Zones also protects your resources from failures in the data center.

Easy and flexible VM Deployments

Managed disks enables you to create up to 50,000 VM disks per Azure subscription per region. Azure VM scale sets are a great way to scale your VM workloads while allowing you deploy thousands of virtual machines in a scale set alone which helps you save money

Secured

Managed disks are designed with security by default which means it is uses RBAC (Role Based Access Control) to protect your data from unauthorized access, the storage are encrypted e.g., Server Side Encryption (SSE) and Azure Disk Encryption (ADE)

Azure Storage Security Best Practice Tips

When we develop and use Azure Storage in cloud development in any platform or programming language, we commonly use SAS (Shared Access Signature) SAS (Shared Access Signature) to access the storage account from your apps or code. However, using SAS for Azure storage access can come with a few major security risks.

For example, if SAS token is compromised, the attacker can access your data. Therefore, it is important to be aware of the best security practices you need to know when using SAS for Azure storage access.

One of the top security recommendations for Azure Storage is to always use HTTPS to distribute SAS, set up some SAS tokens expiration policy, and always use user delegation for SAS whenever it is possible.

Check out a list of recommended storage security best practices and the best practices on using SAS with Azure Storage on Microsoft’s documentation.

Finally, after we read and learned through different types of Azure services for data storage, you should be able to know which one is the best for your needs or business requirements.

Azure Database Services

In the earlier part of this chapter, we learned the common uses and benefits of cloud storage for users and organizations. However, there are still many businesses and organizations that are still having challenges with the data platform strategies with their systems or applications today. Some organizations have outdated data platforms that could use some modernization and upgrades to something better.

There’s been a significant trend of moving existing systems to the cloud, building new applications quickly with the cloud, and offloading some on-premises costs. You need a plan for how to move some workloads to the cloud. And you need to understand how to set up your organization for success. You also need to understand how the role of a database administrator (DBA) or data professional stays the same and what changes you’ll have to make.

Azure SQL as Fully Managed Database Service

Azure SQL Database is a commonly-used cloud database service. It is known to be always up to date and considered a fully managed relational database service developed and built for cloud computing. Using Azure SQL databases, you are able to create SQL databases will a list of good features and capabilities for your applications.

Azure SQL Deployment Options

There are different deployment options available for Azure SQL. Depending on your current workloads and infrastructure, you can choose to deploy your relational databases in different ways like:

SQL Server on Azure Virtual Machines

SQL Server on a virtual machine is considered IaaS. SQL Server on a virtual machine is a version of SQL Server that runs in an Azure VM. It’s just SQL Server, so all your SQL Server skills should directly transfer, though Azure can help automate backups and security patches. SQL Server on an Azure virtual machine is referred to as infrastructure as a service (IaaS). You’re responsible for updating and patching the OS and SQL Server, apart from critical SQL Server security patches.

SQL Managed Instance

The SQL Managed Instance is a PaaS deployment option of Azure SQL that allow organizations to save some money on virtual machines or servers. It helps by giving you the users, an instance of SQL Server but removes much of the overhead of managing a virtual machine. Most of the features available in SQL Server are available in SQL Managed Instance. This option is ideal for customers who want to use instance-scoped features and want to move to Azure without re-architecting their applications.

SQL Database

SQL Database is a PaaS deployment option of Azure SQL that abstracts both the OS and the SQL Server instance away from users. This deployment option allows you to just get a database and start developing applications. SQL Database is also the only deployment option that supports scenarios that require unlimited database storage (hyperscale) and autoscaling for unpredictable workloads (serverless). SQL Database has the industry’s highest availability SLA. It provides other intelligent capabilities related to monitoring and performance, partly because Microsoft manages instances.

History of Azure SQL

In 2008, when Microsoft Azure (previously called Windows Azure), one of its top key components and an important part of it was Micro‐ soft SQL Services. From SQL, Azure was renamed to Azure SQL. Today, both SQL Server and Azure SQL continually evolved with great and better features that even expand to support and work with other open-source databases, PostgreSQL, MariaDB, and MongoDB.

Our modern enterprise systems today are rapidly improving and evolving with modern technologies which also means that in these systems or applications we need to handle and manage large volumes of assorted and diverse data in various formats and types. Therefore, choosing the right data store for your business data is a critical decision.

Check out Microsoft’s important guide in choosing the right data store and the considering data technology choices on the Ready phase recommendation of the Microsoft Cloud Adoption Framework for Azure.

Azure Cosmos DB

Azure Cosmos DB is a fully managed NoSQL database for modern app development. Azure Cosmos DB takes database administration off your hands with automatic management, updates, and patching. It also handles capacity management with cost-effective serverless and automatic scaling options that respond to application needs to match capacity with demand.

Solutions that benefit from Azure Cosmos DB. Any web, mobile, gaming, and IoT application that needs to handle massive amounts of data, reads, and writes at a global scale with near-real response times for a variety of data will benefit from Cosmos DB’s guaranteed high availability, high throughput, low latency, and tunable consistency.

Figure 5-2. An illustration of Azure Cosmos DB

The following are some of the useful benefits of using Azure Cosmos DB for your data solutions:

Guaranteed Scalable Speed

Gain unparalleled SLA-backed speed and throughput for global access, and instant speed and elasticity with SLA support. There is also a support for multi-region writes and data distribution to any Azure region.

Easier App Development

Development is easier and efficient with open source APIs and SDKs supported by Azure Cosmos DB. You have the option to integrate with other Azure services that are commonly used in modern cloud-native app development like Azure Functions, IoT services, Azure Kubernetes Service, App Service, etc. Different programming languages are also supported. For example, SDKs for .NET, Node.js, Java, and Python.

Mission-critical ready

Azure Cosmos DB promises high availability and enterprise-level security that helps the business running. Azure provides the option to easily distribute and replicate your data to any Azure region.

Fully managed and cost-effective

Using Azure Cosmos DB service on Azure is cost-effective and useful as it is a fully-managed cloud service. It is easy to work with database management that you can do on demand using the Azure Portal, Azure CLI, etc. Azure Cosmos DB also support the Serverless (Consumption Plan) pricing model.

Based on the list of features and benefits of Azure Cosmos DB, it is promising and innovative that there are so many ways we can store our data depending on what we really need to build, create and deliver for our businesses.

Learning Different Consistency Levels of Azure Cosmos DB

One of the common confusion of those who are just getting started in Azure Cosmos DB is understanding its different types of consistency levels. It is hard to understand and make sense of it especially when you are not familiar with the concepts of consistency levels, replicas, and throughput in Azure Cosmos DB.

When you are working with Azure Cosmos DB for your databases, it is essential to understand the different consistency levels such as the Strong, Bounded staleness, Session, Consistent prefix and Eventual.

For distributed databases, consistenty is an important concept to understand especially when working with managed cloud databases like the Azure Cosmos DB. Distributed cloud databases are operational across different physical locations. Furthermore, distributed databases depends on replication configuration which usually for the purposes of low latency and high availability, or can be both.

Depending on the priority and requirements, there is a trade-off between choosing the read consistency, availability, latency, and throughput in our distributed cloud databases, as defined by the PACELC theorem.

In terms of distributed cloud databases, the term Replicas can be considered as copies of source data that are the same or identital. These replicas can be hosted in different or another Azure Region. Usually the replication is configured for disaster recovery and geo-based distribution purposes.

If you are replicating your distrubuted databases in the cloud in Azure regions that is located away from your users, there might be possible issues with performance, response time, and latency. Therefore, it is recommended that when configuring the replication, it is ideal to choose the right consistency level for your needs.

Technical details of each consistency levels is beyond the scope of this book; however, if you want to see the comparison and deep dive what is best for your distributed database workloads, I recommend checking out Microsoft’s Guide for Consistency Levels in Azure Cosmos DB.

Azure Cosmos DB integration with Azure Synapse Link

Azure Synapse Link for Azure Cosmos DB is an optional integration that can be used for cloud-native hybrid transactional and analytical processing (HTAP) capability that enables real time analytics over operational data in Azure Cosmos DB.

You can gain cost savings on Azure Cosmos DB using Data Retention with Time to Live (TTL) and the Free Tier. When you use Azure Cosmos DB you do not have to pre-reserve your storage in advance because you get billed for the storage you have consumed directly. However, there are use case scenarios that requires high-write might have an effect on the costs. In order to save costs, you can set a time-to-live (TTL) for your data. This is a time interval after which the data will be deleted. This means that the data that you have already have stored elsewhere can be immediately purged. By doing this, you can you can keep your relevant data updated. Learn more about TTL on Azure Cosmos DB.

Also, you can also make use of the Azure Cosmos DB Tier if you as a developer, your team or your organization are just getting started with the service, or just trying to develop small production application workloads for testing or try out.

The Variety of Azure Cosmos DB APIs

Azure Cosmos DB has different and several options for us users in terms of database APIs. The known database Azure Cosmos DB APIs to choose from includes the Core (SQL) API, Gremlin API, MongoDB API, Cassandra API, and Table API.

These Cosmos DB database APIs helps us to store and manage variety of real-world data such tables, key-value data, graphs, documents, and column-family data models.

The following database APIs enable database engineers, developers and even organizations not to spend so much effort or money on resource in adopting into Azure Cosmos DB. Using these APIs your existing tools, systems, and skillsets can easily get started usingt he same data modelling and querying techniques in respective databases where the existing data are stored.

Azure Cosmos DB APIs are fully-managed service in Azure’s cloud platform. These APIs have features like throughput and storage automatic scaling. It also guarantees flexibility and performance to its users because of its global scale features. The following list are some of the APIs that are currently available for Azure Cosmos DB.

Core(SQL) API

Using Core(SQL) API, you are able to store data in document format and fully manage it with best end-to-end experience with good features and SDK client libraries. The Azure Cosmos DB for SQL API accounts allows database developers and engineers to query database items using the Structured Query Language (SQL) syntax in querying JSON objects. This API is ideal if you are considering to migrate from databases like for example Oracle, HBase, DynamoDB, etc. Also, if you want to modernized technologies to build your apps, SQL API is the recommended option. SQL API supports analytics and offers performance isolation between operational and analytical workloads.

Gremlin API

Gremlin API is useful and worth considering for graph queries and if you have a requirement to store data as edges and vertices. This API is recommended for scenarios involving dynamic data with complex relations. They are too complex that they cannot be modeled with the standard relational SQL databases. The Gremlin API for Azure Cosmos DB combines the features of the algorithms of graph databases with highly scalable, managed infrastructure. This API is based on the Apache TinkerPop graph computing framework and uses the partition strategy of Azure Cosmos DB to do the operations for read and write. A few common use case scenarios where Gremlin API can be useful is for customers graph, geo spatial data, social network graphs, data from Internet of Things (IoT) devices that needs to be represented into graph, etc.

API for MongoDB

The API for MongoDB is specific to MongoDB databases which stores data in a document structure, typically on BSON format. It is compatible with MongoDB wire protocol but not necessarily using code related to MongoDB. Using API for MongoDB is ideal and great choice if you want to work with MongoDB ecosystem. If you already have an existing apps with MongoDB, you can just update the connection configurations and migrate existing data using native MongoDB tools or the Azure Database Migration Service.

Cassandra API

Cassandra API of Azure Cosmos DB is useful if you want to store data written for Apache Cassandra which is usually data in column-oriented schema. This API in Cosmos DB promises a highly distributed API for databases that uses Apache Cassandra. It enables horizontal scaling in storing large volumes of data and offers a flexible approach to a column-oriented database schema. Use this API if your Cassandra databases need to have elasticity and want to take advantage of the fully-managed feature of Azure Cosmos DB. You can use the native Apache Cassandra features, tools, and ecosystem when you use this API, which means that you don’t need to manage the infrastructure like Java VM, garbage collect, etc.Apache Cassandra client drivers are also supported to connect to the Cassandra API.

Table API

The Table API in Azure Cosmos DB stores data in key-value pair format and only supports OLTP scenarios. Any applicationwritten for Azure Table storage can be migrated to the Table API with minimal code changes and can also make use of the premium capabilities.

Azure Cosmos DB and its available APIs give us the flexibility in choosing the appropriate database API for our needs. Let’s learn further about another interesting feature that Azure Cosmos DB offers for distributed database workloads.

Global Distrution and Replication using Azure Cosmos DB

Azure Cosmos DB is a fully-managed and globally distributed database system that allows you to read and write data from the local replication of your databases. It enables replication of data to all the regions associated with your Cosmos account.

It is popular cloud database service in Azure that is intented to deliver elastic scalability of throughput, low latency, well-defined semantics for the consistency of data, and high availability. Thefore, if your application needs fast response time in any location or region in the world, if it’s required to be always online, then building modern applications with on Azure Cosmos DB would be considerably ideal.

A set of databases can be configured to be globally available and distributed in any of the Azure regions. To strategically lower the latency, it is recommended to provision your databases to the Azure Region closes to where your application or database users are. Theforefore, choosing the required regions rely on the global reach of where your users are located. Azure Cosmos DB replicates the data to all the Azure regions that are associated and linked with your Cosmos DB account.

Azure Cosmos DB gives you flexibility by giving you the option to add or remove the regions associated with account when needed. When you update your region for your account, your application doesn’t need to be paused or redeployed. Cosmos DB is available in all five distinct Azure clouds environments available to customers.

Figure 5-4 shows how the distribution of Azure Cosmos DB is replicated and setup globally.

Figure 5-3. Microsoft’s Example of Globe Distribution using Azure Cosmos DB

Azure Cosmos DB Data Migration Tool

If your team in your organization has a need to migrate or import data from external sources into a container in Azure Cosmos DB, I recommend checking out the Azure Cosmos DB Data Migration tool that allows you to import data from sources in file formats as CSV, JSON, XML, SQL, Azure Table Storage, MongoDB, DynamoDB of Amazon, and even from other Azure Cosmos DB API collections.

For detailed information and guide, check out Microsoft’s Documentation Tutorial: Use Data migration tool to migrate your data to Azure Cosmos DB

Choosing an Database API for Azure Cosmos DB

Azure Cosmos DB gives us different options of APIs, which helps us easily migrate our data to Azure. This is regardless if you are trying to build new applications or migrate existing ones. However, every business requirement is unique and different from case to case; therefore, I would recommend you to check this Microsoft guide with a great workflow diagram that can help you with your decision process within your organization.

If you want to know more, I recommend checking this guide Choose an API in Azure Cosmos DB if you and your team are uncertain about which API to choose.

Deploying Azure Cosmos DB as Infrastructure as Code (IaC) is also possible. Azure enables developers and cloud engineers to automate deployments with the feature of being able to integrate and use Azure Bicep in managing Azure Cosmos DB resources. To learn more about how you can do that check this Microsoft’s guide on how you can manage Azure Cosmos DB Core (SQL) API resources with Bicep

Based on what we learned so far, Azure Cosmos DB has great features for any type of data. Choosing the right data store or database in Azure is an important step regardless if you are considering modernizing or upgrade on your existing enterprise systems or currently planning to build a new application hosted on the cloud.

Learn By Doing (Try it out!)

Aside from the supplementary learning hands-on lab repository for the topics in the this chapter, the following quick-start tutorials are most recommended as they are updated based on Microsoft’s technical updates for the service. This section is recommended and optional.

Microsoft Documentation: How to create a storage account

Microsoft Documentation Quickstart: Upload, download, and list blobs with the Azure portal

Quickstart: Build a console app by using the .NET to manage Azure Cosmos DB SQL API account resources

Microsoft Quickstart: Build a .NET Framework or Core application using the Azure Cosmos DB Gremlin API account

Microsoft Learn Module: Explore Azure database and analytics services

Summary

In this chapter, we learned massively about the different types of technologies in Azure for storage and database solutions in the cloud. We learned about Azure Storage which is an important namespace in creating Azure Blob Storage. Aside from the great capabilities of Azure Blob storage, we also learned about the other storage options for IaaS Virtual Machine, such as Azure Managed Disks. Azure File Share is useful in sharing files.

We also learned about the Azure Cosmos DB, its database APIs ideal for different types of databases regardless if they are structured or non-structured. We got some tips and important things to consider when choosing the right API for your Azure Cosmos DB solution, as well as useful resources and links to read further when it comes to choosing the right data storage or database technologies based on your needs.

This chapter has helped us gain knowledge and make better decisions in choosing the right Azure service to use for our applications: existing or new ones to be built or migrated.

Check Your Knowledge

What is the difference between structured and non-structured data types?

What Azure SQL service (PaaS) supports instance-scoped features like CLR and Service Broker?

What free tool can you use to manage your Azure storage locally on your computer?

Why would you consider using Azure Cosmos DB for your data?

What is the difference between Azure Blob Storage and Azure File Share?

To find the answers to the Check Your Knowledge, turn to the back of the book in the Appendix

Recommended Learning Resources

Cloud Academy, 10 Benefits of Using Cloud Storage

Microsoft, Azure SQL for Beginners

Microsoft Learning Path: Implement and manage storage in Azure

Azure Cosmos DB Documentation

Microsoft Documentation: Elements of a Azure Cosmos DB Account

Azure Cosmos DB by Microsoft

TTL on Azure Cosmos DB

Microsoft: Replace your file server with a serverless Azure file share

Azure Cosmos DB free tier

Microsoft Documentation: MongoDB Introduction

¹ Wikipedia, “Microsoft SQL Server”, https://en.wikipedia.org/wiki/Microsoft_SQL_Server

² Bickerton, Peter. 2022. “Why cloud computing is important for data-driven bioscience research.” Earlham Institute. https://www.earlham.ac.uk/articles/why-cloud-computing-important-data-driven-bioscience-research

³ Microsoft Customer Stories, “Sentara Healthcare improves lives for patients, members, and clinicians by centralizing data in Azure, https://customers.microsoft.com/en-us/story/769786-sentara-healthcare-health-provider-azure/

⁴ G2.com Business Software Reviews, “20 Cloud Cost Management Statistics to Help Soar and Save”, https://www.g2.com/articles/cloud-cost-management-statistics

⁵ TechRepublic.com, “10 big data and analytics resolutions for 2022” by Mary Shacklett, https://www.techrepublic.com/article/10-big-data-and-analytics-resolutions-for-2022/

Part III. Artificial Intelligence (AI), Machine Learning (ML), Big Data, Analytics, Internet of Things (IoT) and Security in Microsoft Azure

This third section of the book and its chapters focus on teaching the foundational concepts of artificial intelligence, machine learning, Internet of Things (IoT), Edge technologies, security, and identity management in Microsoft Azure.

Chapter 6 focuses on the useful concepts you need to know about Artificial Intelligence(AI), Machine Learning, Cognitive Services, and Bot Services in Microsoft Azure.

Chapter 7 explores the Big Data, Reporting, and Analytical Services in Microsoft Azure. This chapter will include what you need to know about data analytics, big data, and reporting services in Power BI, Stream Analytics, Data Lake Analytics, Event Hubs, HD Insights, and more.

Chapter 8 explores the different services in Azure for Internet of Things (IoT), Maps, Edge services, and other smart solutions in the cloud for smart things.

Chapter 9 dives into identity and access management(IAM), compliance, cloud security, and DevSecOps in Microsoft Azure which is important in protecting and securing your applications and cloud workloads.You will learn about Azure Active Directory and Azure AD B2C, B2E. Azure security services like Azure Key Vault, Azure Firewal, Azure Sentinel, Microsoft Defender for Cloud and its sub-features, Azure security for networking, data, compute, and security monitoring is also explored.

By the the end of this section of Section 3 you will be able to build intelligent applications using the advanced technologies and services for AI, ML, Big Data, Analytics, and Security in Azure.

Chapter 6. Artificial Intelligence (AI), Machine Learning, and Cognitive Services in Azure

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 6th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at jleonard@oreilly.com.

Making applications easy to use and enabling users to focus on the work that adds the most value will be integral when developing applications going forward. Artificial Intelligence is a central technology for achieving this. Using Azure, all options are covered: a developer can use Cognitive Services to add cognitive intelligence into applications without needing any AI/ML skills. Machine learning engineers and Data Scientists can use Azure Machine Learning in all parts of the project lifecycle. Citizen Developers using low code/no code solutions such as Power Apps can make use of AI Builder to make an app more intelligent.

Håkan Silfvernagel, Manager AI and Big Data at Miles AS, Microsoft AI MVP and Microsoft Certified Trainer

In the previous chapter, we learned about how vital data storage and databases are in cloud computing in Azure and its resources. We learned about cloud storage options for further use and data types, especially in our applications’ processes and operations. In this chapter, we will dive deeper into how we can use the data we have hosted in the cloud through our modern applications in the world of Artificial Intelligence (AI), Machine Learning (ML), and other cognitive services in Azure that will help us build intelligent systems to help us in our daily routines.

Artificial Intelligence (AI) - An introduction

Before diving deep into the vital key and technical concepts of AI and ML and learning about the cognitive services in Azure, we need to know the basics of what they are and how they are helpful to us, our applications, organizations, and society.

AI will reshape our work as technology replaces our mundane tasks and work. Some manual tasks need to be performed by us human beings; however, due to some factors, and reasons, we need to automate, digitilized, and modernize how we conduct our tasks. Using AI as a helpful aid and tool, we can improve our efficiency and effectiveness with our time and the speed of delivery to our business.

As technology rapidly changes, Artificial Intelligence (AI) has become a trending buzzword and continually evolving technology. Even though AI technologies have been seen in many science fiction movies and TV shows, we still have much to learn about this technology. The sci-fi movies we watch also give us the unclear and common misconception that AI and machine learning will take away our jobs and that this technology will replace the human workforce.

Despite these things, we must understand with an open mind and awareness of the value and significance of this critical area of computer science.

AI applies advanced analysis and logic-based techniques.¹., including machine learning.

Source: https://www.gartner.com/en/information-technology/glossary/artificial-intelligence

AI is used to interpret events, support and automate decisions and take action. It is a technical subject that goes far beyond many other areas to comprehend the intricacies and factors involved. Therefore, scientific and mathematical skills are often required when you are working with roles related to AI-related roles in the industry.

Some common programming languages used with AI-related jobs include Python, C#, and R. There are other tools in many platforms today that can be used, including the AI tools available in Azure.

A press release paper released in 2022 by Gartner is included in the following quote:

A press release paper entitled, “Gartner Identifies Three Imperatives Driving the Top Trends in Data and Analytics for 2022”, included in the following quote:

_AI is becoming more pervasive, yet most organizations cannot interpret or explain what their models are doing, resulting in a lack of trust and transparency. Organizations are not prepared to manage the risks of fast-moving AI innovation and are inclined to cut corners around model governance including security, escalating the negative consequences of misperforming AI models, such as incorrect business decisions or worse, those impacting life or death.

As regulations for Artificial Intelligence (AI) proliferates globally, they are mandating certain auditable practices that ensure trust, transparency and consumer protection. By 2026, Gartner anticipates organizations that develop trustworthy purpose-driven AI will see over 75% of AI innovations succeed, compared to 40% among those that don’t._

Source: https://www.gartner.com/en/newsroom/press-releases/2022-03-29-gartner-identifies-three-imperatives-driving-the-top-trends-in-data-and-analytics-for-2022["Gartner Identifies Three Imperatives Driving the Top Trends in Data and Analytics for 2022"]

The quote tells us that AI is the future of technology that continues to help expand our horizons and achieve new cognitive services. On the other hand, because AI impacts people’s lives and our society, we need to focus on doing the work to handle security, deal with the risks associated with AI, and improve performance outcomes.

Today, AI has been used and implemented in application development, cloud development, data science, and more. It reveals that it can take data science and technology to the next level. The AI platform in Microsoft Azure AI platform offers a wide range of services and capabilities to build intelligent applications.

Azure OpenAI Service and Evolution of Chat-GPT

At the beginning of 2023, Microsoft announced to the public the availability of the Azure Open AI Service that enables organizations to build advanced AI models like DALL-E 2, Codex, and GPT-3.5.² The release of the Azure OpenAI promises features that are of enterprise quality and enables the opportunity of creating an AI-powered infrastructure on Azure.

If your organization, developers, or AI engineers want to design and build advanced and optimized AI models, the Azure Open AI Service provides the opportunity for modern AI solutions at a huge scale. A few examples of AI-enhanced tools and technologies are used by some already, including developers. The AI pair programmer GitHub Copilot, PowerApps, and the the GPT-4 integrated Microsoft products like Power BI and Microsoft Designer are enhanced with AI with natural language processing models.

In relevance to the Azure OpenAI service, the Chat-GPT(Generative Pre-trained Transformer), it has gained its popularity, which also caught the interests of Microsoft, the public and other tech giants.³. Microsoft has the vision and mission to make a breakthrough through AI technologies by collaborating with OpenAI.⁴

If you are curious about chatting with Chat-GPT, you can may check it out on this website: Chat-GPT.

Finally, it is important to take note that although the Azure OpenAI Service is generally available, it is actually not open for public use. In order for an organization to use this service, there is a requirement to submit an application to the service. The conditions for following the code of conduct for responsible AI are important once you or an organization get access to use this service for different use cases of development or learning purposes.

Different AI Services in Azure

The AI platform in Microsoft Azure AI platform offers a wide range of services and capabilities that can be used to build intelligent applications. These tools and services for AI and ML in Azure will be discussed in the later part of this chapter.

Examples of AI Technologies and Terms You Need to Know

AI technologies are being used in different fields. The following are some of the known and latest AI technologies you need to know and what they are for.

Natural Language Processing (NLP)

NLP is an intuitive communication between humans and intelligent systems using our human languages. It helps and guides modern interactive voice response (IVR) systems by processing language for communication advancement. For example, a typical application of NLP today is chatbots in websites and systems we use in different businesses and industries.

Internet of Things (IoT) with AI

IoT includes the network of physical objects or devices that have sensors. These sensors are considered useful technology embedded to be able to interact with other devices and collect these telemetries of data for good use. IoT enables these devices to connect and exchange data. This technology has many benefits and uses. For example, IoT is being used in Smart Homes to automate and control heating, security systems, lighting, heating, etc. It is also widely used in different areas like healthcare for health monitoring, transportation to improve traffic management, etc. IoT has the potential to enhance a wide range of industries and bring significant benefits to individuals, companies, and society as a whole.

Computer Vision(CV)

An AI technology that can help us process real-life images through image capturing, processing, and analysis. CV allows machines to understand and gathers contextual, meaningful, and valuable information from the real world. CV techniques have unique requirements for infrastructure and technology that may differ from traditional approaches to Machine Learning(ML). CV is getting better at accurately identifying objects that can help in the development and applications, such as self-driving cars, automated retail stock checks, and automated drones.

Synthetic Data

In machine learning terms, synthetic data is artificially generated and resembles or replicates the statistical properties of actual data without the accurate data’s identifying properties (e.g., id, name, or any personal details). To develop valuable outcomes with this, AI requires an enormous capacity for data. Synthetic data can be generated using various algorithms, such as random number generation, simulation, and statistical models. The goal of developing synthetic data is to provide a representative sample of the data distribution, including the relationships between variables, without exposing accurate data. Some known benefits of using synthetic data in ML include the privacy of sensitive information availability and data consistency for training and testing purposes.

Virtual Agents or Conversational AI agents

A typical virtual agent is a computer application that interacts with us. Many websites provide customer service through chatbot agents to assist and interact with customers to answer their queries. Virtual agents are considered software-as-a-service (SaaS) these days.

Edge AI

An AI technique embedded at the touchpoint where physical devices meet the digital world, Edge AI helps enable IoT. One example of this is that you can also perform a task such as deploying AI and machine learning computing on-premises and to the edge.

Generative AI

A type of AI technology that learns about artifacts from data: after learning from data, it generates innovative new creations that are similar without repeating or copying the original. This is what makes it potentially helpful in creating new forms of creative content like video.

These AI technologies listed are helpful in many ways across industries and are still evolving today. Therefore, the demand for AI professionals working with these technologies is growing.

Why Should You Learn AI?

Development and engineering in the cloud, especially in Azure, involve learning a lot of new technologies and concepts. It is essential to understand the different technologies and their applications in real-life scenarios and business cases. Whether you have a role in leadership, sales or development team, or engineering, learning at least the fundamental concept of AI is necessary.

AI is an essential modern skill to have

Learning AI is a skill that is required and helpful in building modern IT solutions, especially in the cloud. Based on O’Reilly’s Radar report on the state of AI adoption, the most significant barriers are issues with data and a lack of skilled people. The future of work involves automation, and as such, everyone should know the basic concepts of AI. Acquiring such skills will future-proof your career. The rise of AI in the commercial world will also create many job opportunities in various industries. Aside from gaining the necessary skills in AI, there are also some important considerations to think about before implementing AI into any type of business.

High demand of AI-related jobs and careers

Learning AI technologies and tools will help you upskill your competencies in this area and give you better career opportunities. Gaining knowledge and career experience in this field will help you get job roles like being a Machine Learning Engineer, Software Engineer working with modern applications with AI technologies, Data Analyst, Research Engineer, Business Intelligence Developer, and Data Scientist.

Help build better, intelligent, and inclusive applications

Working with AI-related jobs and technologies also allows for making a difference. Since AI can help us build better and more intelligent systems, we can contribute to making our lives and others easier. Also, the more we develop our applications to be more inclusive and human, the better and more used our systems become for daily help and purposes. AI implementation in some industries gives many benefits. For example, virtual assistants and customer service chatbots that we have today. These AI-built tools and services are helping improve our product delivery and customer service satisfaction, which benefits any business or organization, depending on what category it might be.

AI is a flexible and versatile field in tech

AI technology and its use may affect some parts of human life. For example, AI improvements and solutions in industries like customer service, finance, security, fraud detection, and even healthcare can be valuable and helpful. Virtual Health Assistants(VSA) can assist and aid medical patients. VSA helps if the patients are on the correct prescribed medication, makes suggestions on some of the essential health treatments, and may provide ideal diet recipes based on the data history of previous illness and medical records. If you’re interested in learning more, I recommend the role of artificial intelligence in healthcare: a structured literature review medical paper on the role of AI in healthcare.

Benefits of AI to Businesses

AI is a trending topic for companies, especially in e-commerce.⁵. AI’s capabilities are evolving, allowing companies to make a promising approach to customer engagement in real-time, manage their operations, and ensure business continuity, especially when many are working and doing business hybrid and remotely. Organizations are creating new pathways to innovate and expand opportunities along with the advancement of technology.

AI has practical benefits for companies and businesses around the world. Some of the key benefits are:

Increased efficiency and productivity

AI can help automate manual and repeatable tasks and processes. By implementing AI in these manual processes, employees and teams can help focus on more creative and strategic work. It can help increase overall efficiency and productivity.

Better decision making

AI can analyze data and helps provide insights and guidance that can advise better business decisions. It can help businesses to collect, process and analyze large amounts of data. By using advanced algorithms, AI can identify patterns and trends that may not be immediately apparent to humans, and provide insights to help make informed decisions.

Improved customer experience

AI can help companies provide personalized customer experiences, such as customized product or user recommendations or more efficient and effective customer support.

Cost savings

By automating specific tasks, AI can help companies reduce labor costs and increase cost savings. AI can be used to optimize the supply chain, by predicting demand, optimizing inventory, and identifying the most cost-effective shipping routes. By improving supply chain efficiency, businesses can reduce costs and increase profits.

Competitive advantage

Businesses that adopt AI-driven technologies can gain a competitive advantage by helping them make better use of data and quickly responding to changing market conditions.

Predictive maintenance

By using AI in terms of maintenance, it can aid and support companies in predicting when equipment or a machine is likely to fail, allowing for proactive maintenance that can reduce downtime and improve overall operational efficiency.

Fraud detection

AI can assist in detecting fraudulent activity more quickly and accurately, reducing financial losses and improving security.

In general, AI has the potential to revolutionize the way companies do business and bring significant benefits in terms of efficiency, competitiveness, and innovation.

Research about AI in Retail Industry

There is an available infographic report about AI in Retail that was published and forecasted by Juniper Research. This report reveals and concludes the projection of the total retail spending for machine learning in 2023 and the projected annual software spending on AI demand forecasting compared to 2018. This also tells us that spending on ML in AI will continue to grow at a projected rate of 230% between 2019 and 2023. Approximately 325,000 retailers will be using machine learning by 2023, and intelligent checkout technologies in e-commerce or shops will help facilitate 1.4 billion transactions in 2023.

Machine Learning (ML)

ML is a subfield of AI that is focused and specializes in developing algorithms and statistical learning models. These algorithms and learning models enable computers to learn from and predict data. It involves collecting and feeding a massive amount of data into a learning ML model. The model will help identify patterns and relationships within the data and make predictions based on those patterns. The more valuable data the model is trained on, the more accurate the predictions.

There are different types of ML, including supervised Learning, unsupervised Learning, and reinforcement learning.

Supervised Learning

This type of ML involves training a model on labeled data where the correct output is known. The goal is for the model to make predictions on new, unseen data similar to the training data.

Unsupervised Learning

It applies training a model on unlabeled data where the correct output is not known. The model aims to identify patterns and relationships within the data.

Reinforcement Learning

This type of ML consists of training a model to make decisions in an environment by receiving rewards or penalties for specific actions.

ML is a critical technology that enables AI to solve problems. Despite common misperceptions, machines do not learn. They store and compute — admittedly in increasingly complex ways. ML is a purely analytical discipline. It applies mathematical models to data to extract knowledge and find patterns humans would likely miss. ML also recommends actions but does not direct systems to take action without human intervention. Machine learning creates an algorithm or statistical formula (referred to as a “model”) that converts a series of data points into a single result. ML algorithms “learn” through “training,” in which they identify patterns and correlations in data and use them to provide new insights and predictions without being explicitly programmed.

_What are machine learning and deep learning?_ By Gartner, https://www.gartner.com/en/topics/artificial-intelligence

These complex components in an ML system are usually composed of machine learning algorithms, training data, evaluation metrics, business logic, and more. Data engineers, analysts, data scientists, ML engineers, and users are also involved in developing ML systems. It has a range of applications, including recognition for speech and languages. It also includes natural language processing.

The Difference between AI and Machine Learning

AI and ML often get misinterpreted, but they are closely interconnected and related. In a use-case scenario, AI and ML are used together. For example, an AI system is built on techniques like machine learning to create and study the patterns in the training data. Data scientists then optimize these ML models to provide the best results.

MLOps and DevOps: Why Data Makes It Different?

Deploying machine learning projects to production can be challenging because of many factors. In cloud development and software development, we have DevOps. In the Machine Learning, there is also a term called MLOps, which means Machine Learning Operations. There is a good article MLOps and DevOps: Why Data Makes It Different from O’Reilly’s Radar reports on what makes MLOps and DevOps different.

MLOps and DevOps are related and complementary domains that aim to streamline the development and deployment of ML models.

DevOps is a set of practices that focus on teamwork and communication between development and operations teams to automate the process of software delivery and infrastructure management. DevOps helps to increase the speed and quality of software delivery and to make the process more efficient and less error-prone.

MLOps extends the principles of DevOps to the field of using machine learning. It aims to automate the end-to-end lifecycle of ML models, from development and testing to deployment and management. It also helps ensure that learning models can be deployed and managed in production just as easily as traditional software applications.

The practices of both MLOps and DevOps help to guarantee that learning models are developed in a repeatable and scalable way. These practice combinations also help ensure that they are tested and validated before deployment and can be monitored and updated in production. They also help to ensure the reliability and performance of machine learning models in production and to reduce the risk of errors and failures

The article MLOps and DevOps: Why Data Makes It Different from O’Reilly’s Radar reports on what makes MLOps and DevOps different.

Furthermore, must establish a business strategy for AI to identify use cases and success metrics. Some of the typical methods include identifying the benefits of risk reduction, the increased speed of the tasks, the increased number of sales, the increased count of satisfied customers, etc.

We’re just at the beginning of an explosion of intelligent software.

Tim O’Reilly, founder

As Tim O’Reilly said that we are just in the initial phase of evolving and growing intelligent software and applications.

Cloud development in Azure and other cloud platforms will be fundamental in how we do business and work with our innovations and digital products with our customers in the future.

Deep Learning in ML

As a subset of machine learning, Deep Learning focuses on using artificial neural networks with multiple layers to model complex relationships and patterns in data.

It is essential to learn about it because it has revolutionized many areas of AI - from computer vision and NLP to playing popular games like chess and Go. The models for this type of learning have been trained to achieve state-of-the-art accuracy on many tasks and keep pushing the boundaries of what is possible with AI.

In addition, the deep learning in ML is used in many real-world applications, such as autonomous vehicles, speech recognition systems, image classification, and even medical diagnoses. This makes the knowledge of it a valuable skill in today’s job market, as well as a helpful tool for solving challenging problems in a variety of fields.

Deep Learning is constantly evolving, with new techniques and architectures being developed all the time.

Ethical and Responsible AI in Azure

In the future, it has been predicted that robotics and artificial intelligence (AI) are technologies that are prone and suspected to have a vast and essential impact on the improvement of humanity. Important questions are being raised about what humans should do with these systems we teach to think like us and what these learning systems can or should do. The possible risks in letting them do the work for us and how we can effectively manage and control these AI-built systems.⁶

In practical terms, Ethical AI makes sure that initiatives related to AI initiatives of any organization, company, or entity follow and maintain human dignity and should not cause any harm to people or society. It is broad in its spectrum, which means that any AI technologies and products should be built and designed with fairness, liability, safety, and security. One of the commonly known examples and questions raised for ethical AI is the example of a use case of self-driving cars in relevance to handling situations with humans in case of accidents.

There have been many discussions about the ethical implications. Pew Research Center did a research that indicates that even a decade from today, the design of ethical AI design will be widely adopted by many later.

The research, based on a survey conducted by Pew Research Center and Elon University, showed that many are concerned that the evolution of AI will continue to be mainly concentrated on optimizing returns and social authority and that stakeholders will need help to accomplish a unanimity about ethics.⁷

When asked whether AI systems used by organizations that employ ethical principles focus primarily on the public good by 2030, 68% said they will not. The research added that “ethical” implies adopting AI in a transparent, responsible, and accountable manner. For others, it means ensuring their use of AI remains consistent with laws, regulations, norms, customer expectations, and organizational values. Ethical AI also promises to guard against biased data or algorithms, assuring that automated decisions are justified and explainable.

Microsoft has developed Fairlearn, an open-source suite of tools for Responsible AI. Fairlearn is a Python package that helps implement several algorithms to detect and mitigate group fairness issues in machine learning models. It is an essential toolkit for machine learning researchers and developers because, through responsible AI, we would want to train machine learning models and teach the team to behave reasonably. Fairness in responsible AI is essential and can be done in a few ways: fairness towards individuals and groups. It is developed and aims to fix fairness issues in AI.

Microsoft’s Responsible AI Tools and Resources

Microsoft aims to provide education and knowledge and spread awareness of responsible AI to everyone. There are useful learning resources available on resources on website https://www.microsoft.com/en-us/ai/responsible-ai-resources. These resources can help you, your organization, and your team to responsibly use and develop AI at any innovation from concept, development, deployment, and more,

There’s also the Responsible AI Toolbox that will help you gain a better understanding of AI systems more responsibly.

The topic and discussion about Artificial Intelligence are trending but also very important to understand. Most specifically, the importance of Ethical AI and Responsible AI in any design, development, and implementation processes.

Cognitive Services

Azure Cognitive Services is an AI and cloud-based service that helps us build intelligent and innovative applications by implementing cognitive intelligence into our applications.

Regardless of the framework and programming platform type, your Azure services are usually accessible for development using client library SDKs for different programming languages, REST APIs, and platforms built with user interfaces. Because of the variety and flexibility of using these cognitive services and tools, you can be an expert in AI to add and apply cognitive services features to your applications.

Some of the known uses for Azure Cognitive Services include being able to customize pre-trained machine learning models using AI innovation and research. We can deploy cognitive services from a cloud computing platform to edge computing.

Cognitive Services allows you to build cognitive solutions to see, hear, speak, understand, and make decisions. These solutions are grouped into speech, Vision, Language, and Decision.

Figure 6-1 illustrates an overview of the Azure Cognitive Services that are useful and available for developers to use and implement in their applications.

Figure 6-1. An overview of the different Azure Cognitive Services

Table 6-1 reveals some of the commonly used Azure Cognitive Services based on different cognitive categories and how they are useful.

Table 6-1. Different Types of Azure Cognitive Services

Cognitive API Category
Name
Description

Speech

Speech Service

Add speech-enabled features to applications in different ways, such as speech-to-text, speech translation, text-to-speech, and many more.

Vision

Computer Vision

It gives you the ability and access to cognitive algorithms that are advanced and beneficial for image processing and image recognition. If you want to customize and build your image classifications, there is also Custom Vision cognitive service that you can make use of.

Vision

Face Service

This cognitive service gives you access to different face algorithms that help detect and recognize faces.

Language

Language Service

If you need to analyze and understand text, you can use this service with the help of NLP features.

Language

QnA Maker

The QnQ maker service allows you to build a question-and-answer AI service based on the content structure.

Language

Language Understanding (LUIS)

LUIS is a cloud-based conversational AI that uses the intelligence of ML to help predict the overall meaning of a person’s conversational or natural language text. This conversational AI will then be able to process and gather important information from it.

Language

Translator

Translate the messages and text in your application using the cognitive translator service that provides machine-based real-time text translation.

Decision

Anomaly Detector

This AI-based detector will help you detect and monitor the possible abnormalities in time series data. The Azure Anomaly Detector has an interactive demo and example that will help you understand it works. In order to use this demo, you would need an Anomaly resource, endpoint and API key.

Decision

Content Moderator

Using this cognitive decision service you are able to add monitoring for possible offensive, undesirable, and risky content in your application or solutions. Content Moderator is an AI service for moderating content by scanning images, text, videos, etc. and automatically flags to it. This will help build compliance and regulations for your users

Decision

Personalizer

Decision Personalizer lets you select the best possible experience to reveal to your users by learning from their behavior in real time.

These cognitive services help make our applications more intelligent and innovative while helping us save time.

Build Your First Intelligent App with Cognitive APIs

An e-book entitled Building Intelligent Apps with Cognitive APIs (O’Reilly Media) provides in-depth knowledge about building innovative applications using Cognitive APIs.

Important Note on using Azure Cognitive Services

In July 2022, Microsoft made Azure Cognitive Services as Limited Access services to the public to follow consistent Microsoft Responsible AI Principles.

Limited Access services require users to register to use it and only customers managed by Microsoft are eligible for access. For more information, check out Microsoft’s Documentation at Limited Access features for Cognitive Services

Azure Machine Learning

Azure ML is a service for machine learning-related solutions, which helps and aids in improving and managing a machine learning project’s lifecycle. Professionals like data scientists and engineers working with ML or AI can use the available features and tools for designing and developing ML workflow tasks. Such

Tasks related to this field include designing the ML models, training them, and managing the lifecycle of MLOps or Machine Learning Operations.

An ML model can be created using open-source platforms, such as PyTorch, TensorFlow, or scikit-learn. MLOps tools help you monitor, retrain, and redeploy models.

Figure 6-2. Example of typical Azure Machine Learning’s Development Lifecycle

Figure 6-2 shows a simple example of how machine learning models are being developed. As with many processes, you must first define the task. Then explore and prepare your data for a specific purpose or use case. At this point, you’ll be ready to train and validate your model, then deploy it.

Once deployed, you’ll want to monitor and manage your model, ultimately bringing you back to exploring your data and creating a continuous loop. Some workflows have new data pushes every few hours, others every few days. Your MLOps team will determine your needs as they monitor and manage the lifecycle.

The development cycles are created, designed, and developed within a machine learning project involving several professionals in different roles in AI or ML. The development lifecycle for an ML project is often repetitive and iterative. It requires experimentation and analysis of data, machine learning algorithms, and training models.

Like any other project in software engineering and cloud development, Azure Machine Learning cloud service enables its users to work together and collaborate on a machine learning project.

In an ML project, you need to create a workspace called Azure Machine Learning Workspace as a logical container for the project.

A ML Workspace in Azure is a central hub that allows you to manage and collaborate on all your machine learning experiments, models, and assets. It provides a comprehensive set of tools and resources for data scientists, machine learning engineers, and developers to work together and develop high-quality machine learning models.

By utilizing an ML Workspace in Azure, you can:

Collect, store and organize your data

Develop and train ML models

Perform tracking of your machine Learning experiments and results

Deploy ML models to production

Monitor and manage the health of your deployed models

Furthermore, you can also integrate it with other services and tools, such as Azure Databricks, Azure Data Factory, and Azure DevOps, to provide a seamless and collaborative environment for machine learning development and deployment.

Machine Learning Studio

Azure ML Studio is the web portal for data scientist developers. It is a cloud-based IDE that is part of its Azure AI platform. It provides a user-friendly graphical interface that enables data scientists and machine learning practitioners to build, deploy, and manage machine learning models without having to write code.

ML Studio is Microsoft’s central point of contact for machine learning-related computation. This suite will replace the Machine Learning Studio Classic, which will retire in 2024.

Let us check the uses of Azure ML Studio:

Building machine learning models: It provides a wide range of pre-built algorithms that can be used to build models for various tasks such as classification, regression, and clustering.

Data preparation: The studio provides tools for data preparation and conversion, allowing data scientists to quickly clean, process, and format their data before building models.

Model deployment: Once the models are built, the tool provides possibilities for deploying the models as web services that other applications and users can consume.

Model Management: Provides a central location to manage and monitor machine learning models, allowing data scientists to keep track of their models and their performance over time.

Collaboration: The studio enables cooperation and sharing of models and workflows, which helps data scientists to work together on ML projects and share their work with others.

The menu options in Azure ML Studio is divided into three main sections: Author, Assets, and Manage. as described in the following list.

Author is a section in a studio that deals with the creation of the code and set-up of your machine learning processes

Assets are the resources stored and created within the Author section. Assets include the pipelines created in the Designer that controls the resources’ workflow from inputting datasets over a pipeline to the endpoints for the output (e.g., connection to real systems via REST API)

Manage is for the system behind the scenes that include the computation clusters and instances, data stores on which the datasets are stored, and the integrations into other systems

Figure 6-3. Illustration of a ML Workspace creation on Azure Portal

Figure 6-3 illustrates an example of creating an Azure Machine Learning workspace in Azure to get started in creating an ML project.

Overall, Azure ML Studio is a valuable tool for data scientists and machine learning practitioners looking to build, deploy, and manage ML models in the cloud.

Finally, to prepare before the deprecation of Azure ML Studio Classic in 2024, I recommend checking this helpful guide forhttps://github.com/Azure/Azure-Machine-Learning-Adoption-Framework[Azure Machine Learning Adoption Framework] for a basic “lift and shift” migration from the classic version of this ML studio.

Automated Machine Learning (AutoML)

AutoML is perfect for creating models without using code. It’s a No-Code UI, meaning you can follow instructions easily. You can work with machine learning projects without any programming skills required. There is a limitation of this tool. For example, you need to build more complex models.

There are data featurization in Auto ML that are worth checking out. They will help automate machine learning experiments.

AI Builder for Power Platform

AI Builder is a feature in the Power Platform that provides an easy-to-use, no-code platform for building and deploying custom AI models for specific business problems. It is designed to help organizations leverage the potential of AI and ML to automate processes, make predictions, and improve decision-making without requiring specialized data science or AI skills.

With AI Builder, users can select from a range of pre-built templates and models optimized for common business scenarios or create their custom models from scratch. The platform provides a simple and intuitive interface for defining input data and desired outcomes and training, testing, and refining models.

Once an AI model is built, it can be easily integrated into Power Apps, Power Automate, and other Power Platform components, enabling organizations to quickly and easily apply the insights generated by the model to their business processes.

You can use AI Builder in different ways depending on the model you will use. The AI models can be used in the formula bar, add components using AI Builder, and use the Microsoft Power Fx expressions to consume AI Builder models in Power Apps.

The Power Fx is a low-code general-purpose language that is designed and used on spreadsheet-type formulas. It is considered a declarative and strongly typed, and functional programming language, with imperative logic and state control available.

I recommend a good learning path AI Builder on Power Platform if you deep dive further into the details about this tool.

Azure Applied AI Services

Applied AI Services helps you and your organization build and deploy AI-based solutions. Modernize your business processes faster with Azure Applied AI Services. Applied AI Services combine Azure Cognitive Services, task-specific AI, and business logic to offer turnkey AI services for standard business processes.

Automate document processing, improve customer service, understand the root cause of anomalies, and extract insights from any content—all with Azure Applied AI services and extend it using your AI models from Azure ML.

Specialized AI services for specific business scenarios, like modernizing business processes with task-specific AI, can help to solve common approaches. It enables you to accelerate development with built-in business logic that allows you to launch solutions in days rather than months.

Applied AI in Azure is a suite of artificial intelligence and machine learning services provided by Microsoft Azure to help developers build and deploy AI solutions. These services offer a range of tools and capabilities to support the entire AI development lifecycle, from data preparation and model training to deployment and management.

Here are some of the Applied AI services and how they can be valuable for developers:

Cognitive Services

These pre-built APIs allow developers to add features such as NLP language processing, computer vision, and speech recognition to their applications with minimal code. This can help developers quickly add AI capabilities to their projects without building models from scratch.

Azure IoT Edge

The IoT Edge is a platform for running AI models on IoT devices, allowing developers to analyze data at the edge and make decisions in real time. This can be useful for developers working on IoT projects which need to run their models on resource-constrained devices.

Databricks

Databricks in Azure is an Apache Spark-based analytics platform provides a collaborative environment for building and running large-scale machine learning models. It offers a fast and scalable way for developers to process large amounts of data and build models with high accuracy.

Bot Service

Boot Service is a platform for building and deploying conversational AI bots, allowing developers to automate tasks and engage with users in a more natural way. It provides pre-built templates and tools to help developers quickly create and deploy their bots.

Form Recognizer

Form Recognizer uses AI and ML algorithms to extract data from structured and semi-structured records such as invoices, receipts, and contracts. This tool can allow its users to automate the data extraction process from these records, which helps minimize the need for the mundane task of manual data entry. It also supports in improving the accuracy of data. Customized training is also supported in this service.

Cognitive Search

The Cognitive Search is a service that can build AI and ML-powered search experiences within the applications. Some of the common use cases for Cognitive Search services in AI and ML include text analytics, image analysis, NLP, etc. Azure Cognitive Search provides a robust and adaptable platform for building AI and ML-powered search experiences, whether you are a data scientist or a full-stack developer.

Azure Bot Service

Bot Service is a platform for creating, deploying, and operating intelligent conversational bots in Azure. It delivers an integrated environment for developers to build and deploy bots in various channels, such as websites, messaging platforms, Teams, Skype, etc., on the cloud. Bot Service is a platform for creating, deploying, and operating intelligent conversational bots in Azure. It delivers an integrated environment for developers to build and deploy bots in various channels, such as websites, messaging platforms, Teams, Skype, etc., on the cloud. For developers working in AI and ML, the Azure Bot Service provides a convenient and powerful platform to build and deploy conversational bots powered by AI and ML algorithms. The platform offers several built-in features, such as natural language processing (NLP) and language understanding (LUIS), that can be leveraged to build sophisticated conversational bots that can understand and respond to user requests in a human-like manner.

Azure Immersive Reader

The Immersive Reader is a tool in Azure for reading and understanding a text. It delivers a spectrum of components developed to help reading and understanding, including text-to-speech, line-by-line highlighting, and picture dictionaries. It is designed to support people with reading difficulties, including dyslexia, and to help boost reading fluency and comprehension. Across the board, the Azure Immersive Reader is a powerful tool that can help individuals of all ages and abilities improve their reading skills and increase their understanding of the written text.

Azure Metrics Advisor

Protecting the organizations’ growth by enabling them to make the right decision based on intelligence from metrics of businesses, services, and physical assets. Azure Metrics Advisor uses AI to perform data monitoring and anomaly detection in time series data. The service automates the process of applying models to your data and provides a set of APIs and a web-based workspace for data ingestion, anomaly detection, and diagnostics - without needing to know machine learning. Developers can build AIOps, predictive maintenance, and business monitoring applications on top of the service. Furthermore, itso uses AI and ML to deliver anomaly detection and advice capabilities. The service employs advanced algorithms to analyze time series data, identify patterns and anomalies, and provide insights and recommendations for improving the performance and health of systems. Developers can take advantage of it by leveraging pre-built APIs and tools to help them build and deploy AI solutions quicker and more efficiently.

From a broad perspective, using AI and ML in Metrics Advisor enables organizations to improve their systems’ monitoring and performance optimization by helping them recognize and fix potential problems before they evolve into significant issues.

Microsoft’s O’Reilly Book on Azure AI

Microsoft has an e-book available to anyone who wants to learn more about Azure AI. This book is accessible on the Microsoft Azure website and O’Reilly’s platform

Futhermore, the cognitive cervices in Azure follow guidelines and share information about how to responsibly use artificial intelligence in applications. Check out Microsoft’s Guide for Resposible AI with Cognitive Services in different areas.

Also, the Azure AI Infrastructure uses High-performance computing (HPC) that uses a large number of GPU-based computers to solve complex mathematical tasks that help in different problems that we have in our society today.

HPC is frequently used in the following areas: Engineering, Weather Modeling, Finance, Genomics, Simulations, and more. In addition to this, a Software-as-Service(SaaS) solution uses HPC to solve a problem in computer-aided engineering in Azure’s platform.

Learn By Doing (Try it out!)

The following are some recommended quickstart tutorials on Microsoft’s documentation. This section is recommended but optional.

Quickstart: Create workspace resources you need to get started with Azure Machine Learning

Train an image classification TensorFlow model using the Azure Machine Learning Visual Studio Code Extension

Tutorial: Forecast demand with no-code automated machine learning in the Azure Machine Learning Studio

Tutorial: Get started with a Python script in Azure Machine Learning

Azure Cognitive Service for Computer Vision: Quickstart: Optical character recognition (OCR)

Azure Machine Learning Hands on Labs

Set up no-code AutoML training with the studio UI

Summary

This chapter teaches us the basics of AI and how it relates to ML. These two innovations in technology are helpful in many ways. Through AI and ML, we can train models using our training data to build functional AI solutions for our businesses and applications.

As a cloud platform, Microsoft Azure provides a wide range of services to automate creating and deploying ML models. AI in Azure comprises several significant categories: Azure Applied AI Services, Azure Cognitive Services, Azure AI Infrastructure, and Azure Machine Learning. The Cognitive Services provides a set of APIs that developers, data scientists, data engineers, and AI experts in different business application areas and categories can use.

We also learned about the benefits of AI and ML in different areas and how important it is to take considerations of developing AI-driven solutions responsibly by making use of Responsible AI. Microsoft Azure, as a cloud platform and company, is also conscious and responsible for using AI by implementing Microsoft Responsible AI standards that help us ensure that our AI solutions are safe, secure, trustworthy, and ethical.

Check Your Knowledge

What is Artificial Intelligence?

What is the name of an open-source AI fairness project made with Python and is developed by Microsoft?

What is the difference between AI and Machine Learning?

List a few of the Azure cognitive services that help us build and add cognitive features to our applications with the help of AI and Machine Learning.

Why do you think Responsible AI and Ethical AI are important in building AI solutions?

To find the answers to the Check Your Knowledge, turn to back of the book in the Appendix

Recommended Learning Resources

Standford University - Human-Centered Artificial Intelligence, Artificial Intelligence Index Report 2022

O’Reilly Media, AI Adoption in the Enterprise 2021 Report by Mike Loukides

AI Papers to Read in 2022 by Ygor Serpa

O’Reilly Media: Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition by Aurélien Géron

O’Reilly Media: AI and Machine Learning for Coders by Laurence Moroney, ISBN: 978-1-492-07819-7

AI42 Community - An AI Community and School for Everyone

Elements of AI: Free AI Courses in Different Languages

Digital Silk: Artificial Intelligence and Machine Learning

University of Washington: History of AI

Fairlearn: Improve Fairness of AI systems

Microsoft: Responsible AI Resources

Zero to Hero: Azure AI Developer Learning Journey

Technical leaders agree: AI is now a necessity to compete

How Microsoft drives responsible AI

Free e-Book: Mastering Azure Machine Learning

Microsoft’s framework for building AI systems responsibly

Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 3rd Edition by Aurélien Géron

Responsible Machine Learning by Patrick Hall, Navdeep Gill, and Benjamin Cox

Microsoft Whitepaper: Machine Learning solutions with enterprise security and scale

¹ Wikipedia.org, “Logic learning machine”, https://en.wikipedia.org/wiki/Logic_learning_machine

² Microsoft Azure Blog January 16, 2023, “General availability of Azure OpenAI Service expands access to large, advanced AI models with added enterprise benefits”,https://azure.microsoft.com/en-us/blog/general-availability-of-azure-openai-service-expands-access-to-large-advanced-ai-models-with-added-enterprise-benefits/

³ Wikipedia.org, “ChatGPT”, https://en.wikipedia.org/wiki/ChatGPT

⁴ Microsoft News Center, “OpenAI forms exclusive computing partnership with Microsoft to build new Azure AI supercomputing technologies”,https://news.microsoft.com/2019/07/22/openai-forms-exclusive-computing-partnership-with-microsoft-to-build-new-azure-ai-supercomputing-technologies/

⁵ IoTForAll.com, “Artificial Intelligence: Growth Boosting Factor in the New Normal” by Shivbhadrasinh Gohil, https://www.iotforall.com/artificial-intelligence-growth-boosting-factor-in-the-new-normal

⁶ Stanford Encyclopedia of Philosophy, Ethics of Artificial Intelligence and Robotics”, https://plato.stanford.edu/entries/ethics-ai//

⁷ Pew Research Center, Experts Doubt Ethical AI Design Will Be Broadly Adopted as the Norm Within the Next Decade” by Lee Rainie, Janna Anderson, and Emily A. Vogels, https://www.pewresearch.org/internet/2021/06/16/experts-doubt-ethical-ai-design-will-be-broadly-adopted-as-the-norm-within-the-next-decade/

Chapter 7. Big Data, Reporting and Analytics Services in Azure

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 7th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at jleonard@oreilly.com.

From the global economic crisis caused by the pandemic to the rapid adoption of AI tools like ChatGPT, businesses have never been more reliant on data and analytics to compete and stay ahead. To make this happen, legacy silos between data analysts versus scientists or archival versus streaming data must be torn down in favor of an interoperable, governed platform for analytics and reporting. Done well, analytics means getting the right information at the right time to the right people –to do this, the right architecture matters.

George Mount, Founder of Stringfest Analytics, Microsoft MVP, and Author of Advancing into Analytics (O’Reilly Media)

Previously, we learned about how Artificial Intelligence (AI), Machine Learning(ML), and Azure Cognitive Services can be used to build intelligent applications to help us in our daily tasks and work. We explored using these cognitive services to use collected data to develop applications and enterprise-level solutions that will help us in the broader perspective.

In this chapter, we’ll shift gears and dive deeper into how we can use and visualize data collected using cloud services to analyze and understand it better.

Introduction to Big Data, Reporting and Analytics Services in Azure

In our daily routine, it is no secret that we collect, process, and work with data in our applications, mobile phones, tablets, etc. These actions generate data points and understanding that data at scale through extensive data analysis turns datasets into something actionable. This level of analysis is referred to as big data analytics, and it’s the foundation of many data-driven business and enterprise decisions.

By being data-driven as a business or an organization, guessing and misconceptions can be avoided. Visually representing this data to gain insights is helpful in many ways. Before we can transfer our basic information as valuable insights, we need to set up a process for data analysis.

Visually representing this data to gain insights is helpful in many ways. Before we can transfer our basic information as valuable insights, we need to set up a process for data analysis.

Furthermore, every project is unique and can strategically use a different approach. Through cloud computing and advanced technology solutions, we have the opportunity to solve these challenges. We can combine a cloud-based data warehouse with an analysis system that is compatible with the cloud platform. There is also an alternative option to leverage a hybrid workflow by combining cloud-managed services with private clouds.

Before you can transform your raw data collected into insights, you need to configure an analysis process. Each project merits a different approach. Consider factors like how you would prefer to collect and visualize your data. Where would you like to collect this information, and how would your organization want to manage it effectively? How would you like to optimize these enterprise data to improve your services to your users?

A variety of cloud-based data warehouse with a dedicated analysis service. You can combine your big data solutions on the managed services with private clouds or set up your hybrid operation.

What is Big Data?

Big Data is a trending topic in technology today, especially since data is considered valuable. What does working with big data mean? Does it help improve how we handle data collection, reporting, and analysis in our enterprise applications on the cloud or on-premises?

The use case scenarios of big data solutions can be found in many areas. Examples include social media, the Internet of Things (IoT), gaming, etc. In these given examples, millions of users generate large volumes of data through data streams that need to be handled quickly and in real-time. Systems built to handle big data are specialized to solve the challenges of its four Vs. or characteristics. These four V-s are Volume, Variety, Velocity, and Veracity.

Let us deep dive into the details of these four Vs for us to understand them better:

Volume

we generate data every day in different ways, some of the data we generate is of a large volume that needs to be handled differently. Technology platforms used in industries like e-commerce, social media, IoT, gaming, etc., continuously generate large sets of data around the clock. For example, connected cars spread across the globe send live telemetry data to their car manufacturing company’s systems. If we combine all these telemetry data daily, it would be a massive volume of data and can only be solved by big data systems.

Variety

Traditionally, the data we are familiar with and are used to working on are structured data in different types or formats. The data format can be text, time, number, etc., which we commonly collect and store using relational database systems. Increasingly, the evolving demand and use of unstructured data for things like live streaming of media files, images, feeds on Twitter, websites, live logs, etc., makes it challenging to have a data analytics system to analyze all these data types. This data variation is one of the challenges big data systems solve.

Velocity

A velocity of data often refers to how high the speed of incoming data processing is. As mentioned earlier, gaming or IoT systems that require speed of delivery and data processing are good examples of the need for velocity.

Veracity

The veracity or integrity of data is an essential aspect of big data. It is because it checks and verifies data authenticity, trustworthiness, precision, and accuracy. Veracity is vital mainly when the data to be analyzed will be used in making critical analytical decisions. The accuracy and reliability of data are crucial for trustworthy output or results.

Additionally, these challenges can be solved in a cost-efficiently way using big data systems. Today’s big data technology platforms use distributed processing for scalability, fault tolerance, and high throughputs.

In distributed processing, multiple computing resources work together to execute a task. Hadoop is a big data solution built on distributed processing concepts. It can handle computer nodes that can work together up to tens of petabytes (PBs)¹ of data to perform a query task.

For example, Azure HDInsight (a managed Hadoop cloud service) and Azure Databricks are widely used big data platforms on Microsoft Azure. We will learn more about the different big data services and solutions throughout the rest of this chapter.

Figure 7.1 shows an example of the typical data flow process in Big Data which involves data collection, storage management, then big data analytics, where a decision-making unit is critical in moving further to the network optimization unit.

Figure 7-1. An example of a data flow process in Big Data with different types of units

Big Data Solutions in Azure

Cloud services for Artificial Intelligence (AI), Big Data, and Analytics are part of the cloud platform Azure provides to its users. The big data solutions and services in Azure give an advantage and flexibility to those users who want the synergy of combining cloud computing capabilities with analytics.

Regardless of if they are structured or unstructured data or if they are data of high volumes that require colossal storage capacity. Azure Big Data services come with fully managed infrastructure and real-time analytics. In addition to this, they can also be well integrated with other cloud resources in Azure. Examples include integration with other analytics, machine learning, storage, and database services.

Aside from computing, networking, hybrid, and other cloud solutions, Microsoft Azure also provides a wide range and a list of services that can help you set up a significant data infrastructure. Including databases, data processing or management tools, analytics, etc., to work and manage complex data sources.

Use Cases of Big Data in Azure

Aside from computing, networking, hybrid, and other cloud solutions, Azure provides a range of services that can help you set up the infrastructure for big data. These include databases for data processing and analytics for machine learning and integrating complex data sources.

The following are some of the everyday use cases of using Big Data in Azure:

Databases

The options for database management include Table Storage, self-managed databases hosted on virtual machines, and other managed databases. Examples of these types of databases available are SQL Server, PostgreSQL, MySQL, and MariaDB. There is also an option for self-managed Table Storage. You can take use of Azure Cosmos DB for distributed database that is fully-managed . Cosmos is a globally deployable and replicable service that is scalable, adaptable, and low latency. It also supports several of different database engines. Its application programming interfaces are interoperable with a wide variety of tools. Some of these include Cassandra, SQL, Apache Spark, MongoDB, Jupyter, and Gremlin. Azure also includes SQL Data Warehouse, designed for large-scale structured data; Azure Data Lake, designed for large-scale structured data; and Azure Data Lake, is designed for large-scale structured data, and Azure Data Lake, which is designed noSQL type of data.

Analytics

Azure delivers its customers an extensive variety of data analysis products and services of data analysis products and services to its customers. HDInsight and Azure Analysis Services are two examples of the broad benefits. It provides a business-grade analysis engine that can aggregate data gathered from a variety of sources and transform it into a semantic business intelligence model that can be accessed with ease. The service can generate interactive dashboards and reports and integratein addition to integrating previously defined data models.

Data Engineering

You can use two primary Azure services to create complex data pipelines: Azure Data Factory and Azure Data Catalog. Data Factory provides serverless integration for local and cloud-based data repositories. These two category of data engineering tools in Azure will be discussed in more detail in the next section.

Machine Learning

Azure provides various solutions for artificial intelligence and machine learning, including Azure Machine Learning Services (AMLS). The AMLS lets you create customized models for machine learning without the programming skills required by using a zero-code drag-and-drop interface and a code-first environment. It is compatible with open-source tools and platforms such as PyTorch, TensorFlow, Open Neural Network Exchange(ONNX), and sci-kit learn in Python. Azure Machine Learning Services helps automate machine learning with tools like automated feature selection, algorithm selection, and hyperparameter scanning.

Complex Big Data Pipeline Tools in Azure

Big data can be complex to visualize and analyze; sometimes, you must create pipelines to handle them. In Azure, two big data solution tools would help complex big data pipelines. Let us briefly overview these tools: Azure Data Factory and Azure Data Catalog.

Azure Data Factory Pipelines and Integration

The Azure Data Factory allows serverless integration for both cloud-based and local data repositories. Using Data Factory, you can perform tasks for extract, load, and transform (ELT) using the available data connectors provided natively by Azure. Some built-in connectors in Data Factory for data sources are from Google Big Query, Amazon S3, and many other data sources from other platforms and on-premises.

Furthermore, transferring or copying existing data from Azure File Storage to Data Factory is possible. This service in Azure allows you to build ETL through a visual editor, which gives flexibility to those who prefer to do it without writing code or scripts. The tool for Data Factory enables you to automate the process with scheduling, drag-and-drop components, and even create event-based triggers.

Azure Monitor is also an excellent monitoring service that can be well integrated with Data Factory. It helps you gain the overall view and visibility to manage the performance of data flowing through CI/CD pipelines.

Azure Data Catalog

Data Catalog is a fully managed metadata catalog for enterprise use. It supports a record for metadata that helps in data asset discovery. It allows you to crowdsource metadata and annotations to users like data analysts, engineers, or data scientists.

It provides a way to enable these types of users to discover, understand, register, enrich, and consume data sources. Data professionals can share their knowledge and collaborate, which can help make data more easily searchable and accessible.

You can handle complex big data pipelines using Data Factory and Data Catalog. However, it is worth to take note that Azure is replacing Azure Data Catalog with the service by 2025. If your organization has existing Azure Data Catalog accounts, you need to start planning your migration to Microsoft Purview.

Building, Configuring and Deploying Big Data on Azure

Microsoft recommends a four-step process for designing and developing a new big data solution on its platform. These processes include evaluation, architecture, configuration, and production.

Evaluation of a Big Data Goal and Solution

The first step is to evaluate the goals and purpose of big data goals. If you are an organization that is new to this journey, you need to evaluable and understand what kind of data you want to include in your planned solution and what you want it.

For example, web scraping data from the web completely differs from the data telemetry gathered from an IoT device. The amount and type of data used will assist in planning for data ingestion and identifying what kind of storage is required; once you know what type of data needs to be processed, a decision on how to analyze it must be made.

If an organization needs a dedicated big data scientist, then they can use these available big data services available to help them get started. Another good option is to make use of machine learning (ML) within the system would be helpful.

Moreover, if your organization is nevertheless in the starting process of your cloud journey, it is recommended to familiarize yourself with the full spectrum and strategies needed for an Azure migration. I recommend starting an intelligent and effective by considering to start your project by initially migrating the core applications and processes to the cloud.

Identifying Big Data Architecture

Big data architecture can be classified into different categories based on various factors, such as data sources, processing requirements, and storage mechanisms. Let us check on some of the common ways to classify big data architecture:

Data Volume

Big data architecture can be classified based on the size of the data. There are three types of data volume: small, medium, and large. The architecture used for each type of data volume varies based on the amount of data.

Data Velocity: Big data architecture can be classified based on the speed at which the data is generated and processed. There are three types of data velocity: batch processing, near-real-time processing, and real-time processing.

Data Variety

This can be classified based on the variety of data types that are processed. There are three types of data variety: structured data, semi-structured data, and unstructured data.

Data Sources

This can be categorized based on the sources of data. There are two types of data sources: internal data sources and external data sources.

Processing Requirements

A big data architecture can be classified based on the processing requirements of the data. There are two types of processing requirements: simple processing requirements and complex processing requirements.

Storage Mechanisms

The classification can be based on the storage mechanisms used for the data. There are three types of storage mechanisms: distributed file systems, NoSQL databases, and Hadoop Distributed File Systems (HDFS).

Deployment Model

An architecture for big data can also be classified based on the deployment model used, such as on-premises, cloud-based, or hybrid.

By considering these different factors, it is easier to classify big data architecture based on its characteristics and requirements. The solutions for Big data usually involve one more workload type, like batch processing of big data sources at rest, real-time processing of big data that are in motion, interactive exploration of big data, machine learning, and predictive analytics.

Therefore, a typical big data architecture should be able to handle complex or large amounts of data for ingestion, processing, and analysis that traditional database systems and applications cannot take. The Big Data Architecture Style for Azure would help you and your organization gain insights as you plan to design, implement and build your solutions for big data.

Preparation of Production Environment

After choosing the services, you need, you can configure and prepare your production environment. Your exact configuration will depend on your selected services, the data sources, and the environment type.

Based on your structure, you typically monitor the processes as often as possible in order to get the best performance and return on your investment.

Azure Monitor and Log Analytics in Azure help many organizations prepare and monitor their workloads, resources, and data for production use.

By monitoring the workloads and applications, creating privacy and security policies, and implementing disaster recovery for your big data system, you are fully utilizing the capabilities you need to be successful in your big data solutions. Based on these processes and guidance in building a big data solution, you should be able to make a solution ready to use in the cloud.

Data Analytics - Making Sense of it and its Importance

Data analytics is data collection, extraction, transformation, and modeling. The purpose and objective of it is to collect and identify useful information to assist with business decision-making. We are reminded of number comparison and spreadsheet-based data analysis when considering analytics.

The process of performing analysis of data is a relatively fresh and gaining prominent field. The origins of it can be traced back in the 20th century when businesses began collecting data to enhance their operations. Due to the high cost of storing and analyzing data, it could have been utilized more effectively.

In that century, the development of computers and the internet enabled the accumulation and analysis of vast data, leading to data analytics. In the 1980s, data warehousing and business intelligence instruments facilitated businesses’ efficient collection and analysis of data. In the 1990s, the rise of the Internet and e-commerce led to the accumulation of vast data, necessitating the development of new analytical tools and techniques.

With the growing significance of data in business, data analytics has become a crucial field of study. It enables businesses to make informed decisions by identifying patterns, trends, and insights through data analysis. Data analytics has become indispensable for identifying trends and making predictions in many other sectors, including healthcare, education, and government.

Today, analytics is an essential component of many enterprises and organizations. Increasing accessibility to data and technological advancements have facilitated the collection, storage, and analysis of vast quantities of data.

Therefore, data analytics has become indispensable for businesses and organizations seeking a competitive edge.

The Different Types of Data Analytics

Most companies are likely already using some analytics solution. Businesses are increasingly embracing sophisticated data analytics solutions with machine learning capabilities to make more reasonable business decisions. Analytics is also used to help determine market trends and opportunities.

Many organizations that do not start to use data analytics with bold, future-casting capabilities may uncover business implementation lacking because they cannot find hidden patterns and achieve other insights.

Let us take a look and understand the different types of data analytics that will help you in your big data and analytics project.

Predictive Data Analytics

It is predictive analytics is the most utilized type of data analytics. This form of analytics employs statistical models and algorithms of machine learning to analyze past data and predict future outcomes. It assists organizations in anticipating future trends and behaviors and making decisions based on these predictions. This form of analytics is utilized by businesses to determine trends, correlations, and causes. The classification can be separated into predictive and statistical modeling, but it is essential to understand that both of these go hand in hand. For instance, an advertising campaign for t-shirts could use predictive data to determine the degree to which the conversion rate correlates with the target audience’s geographic location, income bracket, and interests. Then, predictive modeling could be utilized to analyze the statistics for two or more target groups. Then, provide revenue estimates for each demographic.

Prescriptive Data Analytics

Prescriptive analytics is where AI and big data are combined to help predict outcomes and identify what actions to take. This type of data analytics is composed of random testing and optimization. Using advanced features in Machine Learning (ML), prescriptive data analytics can help answer questions such as the best possible solution or action in decision-making. With the help of ML, you have the option to test and verify the correct variables and recommend new ones that are more likely to generate a positive outcome.

Diagnostic Data Analytics

Diagnostic data analytics may not be as exciting as predicting the future; however, analyzing past data can be essential in guiding your business. Diagnostic data analytics is examining data to understand the cause and event or why something happened. Techniques such as drill down, data mining, correlations, and data discovery are often employed. This type of analytics usually helps detect and answer why something occurred.

Descriptive Data Analytics

Descriptive analytics is the backbone of reporting. Business intelligence (BI) tools would not be possible without the descriptive data analytics. It helps answer the essential questions like “how many, when, where, and what” of any data. There are two categories of descriptive data analytics: canned reports and ad hoc reports

Data analytics can help in many ways and is essential because it helps businesses optimize their performances. Implementing data analytics strategies in companies or organizations can help reduce costs by learning and identifying possible and valuable ways of doing business. Data analytics help organizations make better business decisions by analyzing user trends, customer satisfaction data, etc., which helps create better services and products.

Azure provides a combination of analytics services. The HDInsight and Analysis Services are two of the common ones. For example, the analysis s has capabilities for enterprise-grade analysis. It helps with the data collected from many data sources. It can also convert these collected data to a usable semantic model for the business intelligence (BI) model. Furthermore, you can also use this to integrate with your pre‐ defined database models and create reports and dashboards that can be customized.

On the other hand, the Azure HDInsight is a cloud-based Big Data analytics service helps in the analysis and process huge capacity of data using popular open- source technologies for big data including Apache Hadoop, Spark, Hive, HBase, and Storm. HDInsight provides a highly scalable platform for Big Data analytics that can quickly scale up or down based on your processing requirements. This means you can process large volumes of data without worrying about infrastructure constraints.

It is a cost-effective solution for Big Data analytics since you only pay for the processing power and storage you need, and you don’t need to worry about the underlying infrastructure costs. Another good feature of this service is that it gives you the flexibility to design and develop solutions that require multiple open-source Big Data technologies supporting a variation of tools, frameworks, and programming languages.

The Azure HDInsight delivers enterprise-grade security components such as authentication, authorization, and data encryption. These features and capabilities help to safeguard your data and strengthen compliance with your organization’s regulatory requirements.

Azure Big Data Services

Azure offers a suite of cloud services for Big Data solutions and development that enables and helps organizations with their operations and explore large volumes of data.

Figure 7-2 shows an example of a big data architecture designed to help handle huge and complex data of traditional database systems and applications. The big data architecture allows ingestion, processing, and analysis of data. The recommended architecture for big data comprises several components like data sources, data storage, real-time ingestion, batch processing, and stream processing with machine learning to reporting.

Figure 7-2. An example of a big data architecture style and flow with examples of Azure services

Azure includes many benefits that can be used in a big data architecture in two categories:

Managed services that include the Azure Data Lake Store, Azure Data Lake Analytics, Azure Synapse Analytics, Azure Stream Analytics, Azure Event Hub, Azure IoT Hub, and Azure Data Factory

Open-source technologies based on platforms like Apache, Hadoop, HBase, Hive, Pig, Spark, Storm, Oozie, Sqoop, and Kafka. These technologies are available on Azure in the Azure HDInsight service

Azure Data Lake

The Data Lake Storage in Azure is a scalable and secure cloud-based data storage service that lets organizations store and process big data in its native format. It provides enterprise-grade security features such as encryption, authentication, and authorization and integrates with popular Big Data tools like Hadoop and Spark.

Azure NoSQL for Big Data and Analytics

NoSQL databases are highly scalable and provide integration support to various applications and workloads. The NoSQL databases are popular alternatives to traditional databases and drive robust support from cloud vendors like Azure.

Big data storage for NoSQL databases can use Azure Cosmos DB and Hbase on HDInsights. Additionally, there is also a fully-managed data exploration service like Azure Data Explorer that can also be used for analytical databases

Azure Stream Analytics

Stream Analytics is a real-time stream processing service in Azure that allows organizations to analyze and process streaming data from various sources. This real-time data can be streamed from IoT devices, social media channels, and applications. It can help its users get real-time insights that can help organizations make better decisions.

Data streams with an example shown in Figure 7-3, are being used in data analytics and data engineering to enable sensors, applications, Internet of Things (IoT), gateways and smart monitoring devices to gather and broadcast real-time continuous event data known as data streams. Streaming data is high volume and has a lighter payload than nonstreaming systems.

Data engineers use Azure Stream Analytics to process streaming data and respond to data anomalies in real time. You can use Stream Analytics for Internet of Things (IoT) monitoring, web logs, and point of sale (POS) systems.

Figure 7-3. An overview of the different Azure Stream Analytics in action with use case and Azure services

Stream Analytics is useful in scenarios where your organization must respond to real-time data events or analyze large batches of data in a continuous time-bound stream. Or your organization must choose and decide whether to work with streaming or batch data. Data ingestion can happen from applications and gateways into an event hub or IoT hub. The event hub or IoT hub then streams the data into Stream Analytics for real-time analysis.

Batch systems process groups of data stored in an Azure Blob store. They do this in a single job that runs at a predefined interval. Don’t use batch systems for business intelligence systems that can’t tolerate the predefined interval. For example, an autonomous vehicle can’t wait for a batch system to adjust its driving. Similarly, in real-time, a fraud-detection system must decline a questionable financial transaction.

Azure Synapse Analytics

Synapse Analytics (formerly Azure SQL Data Warehouse) is also another type of analytics service that provides a unified experience for big data and data warehousing. What it does is that it mixes the data warehousing and big data as a single unit of service. Doing this gives the capabability to organizations to analyze and query large volumes of structured and unstructured data using SQL, Spark, and Power BI.

Using Synapse Analytics, you can integrate these two and it allows you to query data on your terms, using serverless or dedicated options—at scale. Azure Synapse combines these worlds with a unified experience to ingest, explore, prepare, transform, manage, and serve data for immediate business intelligence (BI) and machine learning needs.

Furthermore, Azure Synapse Analytics is the next Azure SQL Data Warehouse generation. It enables you to transform and load from any data source – both relational and non-relational databases. Regardless if you are doing this on-premise or in Azure, it unifies all the data and lets you process and analyzes it using the SQL language. Azure Synapse Studio is an excellent tool to use for creating a workspace.

Azure Databricks

Databricks is a quick, manageable, and combined Apache Spark-based analytics platform in Azure. It delivers a collaborative workspace for data engineers, data scientists, and data analysts on different projects. With Azure Databricks, organizations can efficiently process and analyze big data using Apache Spark, Python, and R, and build and deploy machine learning models at scale.

Using this service also lets you set up managed Apache Spark clusters with auto-scaling and auto-termination, eradicating the complexity of setting up Spark in your regional data center. Figure 7-4 illustrates an example of implementation of Azure Data Bricks.

Figure 7-4. An example illustration of Azure Data Bricks implementation

Azure Data Lake Storage

Azure Data Lake Gen2 is designed for enterprise big data analytics with dedicated features and capabilities. It is built on Azure Blob Storage that includes the features and elements of Azure Data Lake Storage Gen1.

The Data Lake Storage Gen2 promises some good features that benefit security, performance, and management. It supports file system semantics, file-level protection, and scaling options. Since it is built on top of Azure Blob Storage, there are flexible options for storage types in different tiers. There are also low-cost options and support for high availability and disaster recovery.

Azure HDInsight for Hadoop, R Server, HBase, Spark and Storm Clusters

Over the course of time, the free and open-source computing platform known as Apache Hadoop has been one of the most factors in the field of big data. Hadoop’s ecosystem is still growing and developing, despite the reality that Hadoop itself is not very well known. It also offers several compelling application cases. It allows you to carry out complicated, distributed analysis operations on practically any volume of data.

Azure HDInsight is an open-source, managed analytics and cloud-based service for users that require more significant and broader support for analytics capabilities for big data. Organizations that need to process vast quantities of data with streaming or historical data capabilities could take advantage of this.

Using HDInsight, you can create significant data clusters using Hadoop and scale them based on your demands and workloads. It allows you to implement Hadoop analytics for existing data by integrating with other services and tools like Azure Data Lake Storage and the Azure Data Factory.

The support for Hadoop benefits those who need to use the capabilities of Hadoop tooling like Hbase, Apache Spark, Storm, Hive, and Apache Kafka.

In addition, Azure HD Insight also provides monitoring, high availability, security, and compliance at an enterprise scale in Azure. You will benefit from Azure HDInsight if you utilize the custom code to process and analyze big datasets with the latest frameworks available such as Spark, Hadoop, HBase, Hive, or Kafka.

The HDInsight also gives you control over your cluster setup and the software installed on them. HDInsight is also a recommended alternative if you migrate Hortonworks, Cloudera, or MapR collections from on-premises environments or other clouds.

Overall, Azure HDInsight can be used for a variety of scenarios for big data. These data types can be historical or real-time data used for data warehouses, data science, IoT, data science, and hybrid data solutions.

Azure Data Factory

Data Factory is a data integration service on Azure. It enables developers to create, schedule, and manage data pipelines capable of moving and transforming data from various sources and destinations. Developers can perform data integration duties such as extraction, transformation, and loading (ETL) from on-premises and cloud-based data sources, including SQL Server, Oracle, MongoDB, Salesforce, Azure Blob Storage, and others. The service is compatible with Hadoop Distributed File System (HDFS), Apache Spark, and Microsoft Azure HDInsight.

Developers can construct data pipelines using Data Factory’s user interface or code-based authoring tools such as JSON, Python, and . NET. Once the data pipelines have been constructed, they can be scheduled and monitored to ensure the data integration processes operate efficiently and reliably.

Data Factory offers a variety of integration options with other Azure services, such as Azure Functions, Azure Logic Apps, Azure Stream Analytics, Azure Databricks, and more, to enable the creation of complex data workflows capable of handling real-time data streaming and data orchestration scenarios.

Azure Analysis Services

Analysis Services is an enterprise-grade data model in the cloud. It is ideal for an enterprise that needs an analytics engine as managed cloud service. It can be configured using infrastructure as code tools like the Azure Resource Manager (ARM template), Azure Bicep, and Terraform. You have the option to combine data from multiple sources and build them as one semantic model.

It allows you to develop high-performance Business Intelligence (BI) solutions with secured access and fast time to delivery. It can be configured to perform autoscaling based on the analytical workload, and you pay only for the resources you consume. Analysis Services also lets you import existing tabular models.

Figure 7-5. An overview of the different Azure Analysis Services

In Figure 7-5, you will find an overview of the different kinds of solutions and tools used for analysis in Azure.

Power BI Embedded Analytics

Microsoft Power BI is a leading data analytics service in Azure. Many people are familiar with Power BI because they interact with reports in the Power BI service or the Power BI for mobile apps. This service enables many options for different kinds of roles. Those working with data analytics and reporting can use Power BI efficiently.

In addition, cloud engineers and software developers can use any Power BI content embedded in their applications by programming it. Developers can programmatically embed Power BI analytics in applications to visualize and present data or reports. The end users of the Power BI embedded analytics can easily make view the actual data and reports and can easily make decisions based on facts and statistics, as shown in Figure 7-6 which illustrates of Power BI Embedded Analytics.

You can embed Power BI content in any app by using an HTML iframe element. It is purpose-built for developers, and it can be used with client APIs REST APIs; there are SDKs for different programming languages and platforms, such as .NET, JavaScript, and TypeScript.

Figure 7-6. An illustration of PowerBI

Suppose you are a developer who wants to learn more about how you can enrich your data analytics and reporting using Power BI embedded analytics. In that case, check out Microsoft’s Guide on How to Embed Power BI into web application or Portals.

Microsoft Purview for Data Governance

Microsoft Purview (previously Azure Purview) is a solution in the governance portal that provides a suitable data governance service that helps users to manage their on-premises, multi-cloud, and software-as-a-service (SaaS) data.

It helps automate data discovery through data classification and scanning as a service for assets across your data estate. Once the metadata and descriptions of the discovered data assets are, they can be integrated into your data estate’s map. Purpose-built apps create environments for data discovery, access management, and insights.

Microsoft has documented Microsoft Purview account architectures and best practices that will help with centralized your data governance solutions. Additionally, the Microsoft Purview governance portal also provides features for governing data, such as the Data Catalog, Data Estate Insights, Data Sharing, and discovery challenges for data consumers.

Try Free Premium Versions of Microsoft Purview solutions

If you want to try the different solutions and features of Microsoft Purview, a 90-day Microsoft Purview solutions trial is available. It is a good option for the organization to explore the Azure Purview capabilities for compliance needs. Microsoft 365 E3 and Office 365 E3 customers can start now at the Microsoft Purview compliance portal trials hub.

Microsoft Purview Data Lifecycle Management provides the features and tools necessary to retain the content that you require to keep and erase the content you don’t. Frequently, regulations and compliance standards demand that content be managed and deleted. However, eradicating content lacking business value may assist in risk and liability management. For instance, it reduces compliance and security risks.

Final Note on Data Management and Analytics in Azure

A final note worth highlighting is that organizations that desire and want good quality analytics should work with accurate, trustworthy, and robust data. Data and information security practices are highly recommended to ensure your enterprise data is secured both in transit and at rest.

Azure Data Encryption at rest and using different types of data encryption models are a few of the tools offered by Microsoft Azure to protect and secure your enterprise data according to your organization’s data security and compliance requirements..

Learn By Doing (Try it out!)

Aside from the supplementary learning hands-on lab repository for the topics in this chapter, the following quickstart tutorials are most recommended as they are updated based on Microsoft’s technical updates for the service. This section is recommended and optional.

Azure Synapse Analytics for data engineers

Microsoft Quickstart: Share and receive Azure Storage data in-place with Microsoft Purview Data Sharing

Quickstart: Create an account in the Microsoft Purview governance portal

Analyze data with Apache Spark in Azure Synapse Analytics

Create Apache Hadoop clusters using the Azure REST API

Summary

In this chapter, you earned about the fundamental concepts of Big Data, such as its characteristics, common uses and benefits, and its importance to data science, analytics, machine learning, and reporting. We also learned about Azure’s different big data, analytics, and reporting services.

Also, we learned about Microsoft Purview’s solutions in the governance portal provides a data governance service that helps you manage your on-premises, multi-cloud, and software-as-a-service (SaaS) data.

Check Your Knowledge

Big data, data science, and analytics are evolving and trending. Why do you think they are essential?

Big data and analytics tools without good cybersecurity protection could put an organization at security risk and vulnerable to data breaches or cyberattacks. What do you think an organization can do as precautions to prevent this from happening?

What are the common challenges in building big data solutions?

Apache Hadoop is used in some big data analytics solutions. Why do you think Hadoop is commonly used in big data analytics?

How does Azure Synapse Analytics differs from Azure Data Lake Analytics?

To find the answers to the Check Your Knowledge, please turn to back of the book in the Appendix

Recommended Learning Resources

Microsoft’s Cloud-Scale Analytics

Microsoft Documentation for Azure Analysis Services

Microsoft Purview Data Governance Documentation

Hands-On Data Warehousing with Azure Data Factory by Christian Cote, Michelle Gutzait, Giuseppe Ciaburro

Video: Azure Analysis Services Overview to learn how Azure Analysis Services fits in with Microsoft’s overall Business Insights capabilities

O’Reilly Book - Stream Analytics with Microsoft Azure

Video: The future of data governance: Introducing Microsoft Purview

Microsoft Documentation: Big data Architecture style

Microsoft Learn: Introduction to data analytics on Azure

Data Science for Beginners

Modern analytics architecture with Azure Databricks

Pluralsight Course: Data Science with Microsoft Azure

Data Exposed: Big Data Cluster Security

Video: Azure Analysis Services Overview

¹ Forbes.com, “How Do We Define Big Data And Just What Counts As A Big Data Analysis?” by Kalev Leetaru, https://www.forbes.com/sites/kalevleetaru/2019/01/09/how-do-we-define-big-data-and-just-what-counts-as-a-big-data-analysis/

Chapter 8. Cloud IoT (Internet of Things) and Maps Services

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 8th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at jleonard@oreilly.com.

The Fourth Industrial Revolution, or Industry 4.0, introduced smart automation and increased interconnectivity in the industry. IoT technologies are a great enabler for Industry 4.0, and Azure cloud allows you to build secure, compliant, and scalable IoT solutions.

Goran Vuksic, Engineering Manager at Pandora, Microsoft AI MVP

In the earlier chapter, you learned about the different solutions and services to plan, design, and build solutions for Big Data, Analytics, and Reporting in Azure, which relates to the topic of this chapter. Your knowledge of analytics and reporting will apply to handling the metrics and telemetry data gathered from IoT devices for the cloud.

In this chapter of the book, we will learn about the IoT, maps, and edge services in Azure.

After the essential topics in this chapter, you will understand the Azure services you can use in different use cases to develop IoT solutions.

Internet of Things

Today’s architectures feature highly scalable event handling, which enables real-time analysis in what Microsoft has named the Intelligent Cloud. They support the deployment of machine learning at the Intelligent Edge in the devices.

As more advanced IoT solution components and capabilities have become available, previous architecture patterns have evolved to take advantage of these new capabilities and enable the deployment of more sophisticated business solutions.

IoT devices are gaining in popularity and are evolving. IoT devices such as smart lamps, smart wearables, and temperature sensors are becoming essential to our daily lives and routines.

Smartwatches help us get moving and exercise more efficiently by giving us data results and statistics on how we perform. Smart homes and lamps help us control our home and office lighting and appliances without manual work. Temperature sensors allow us to monitor the temperature of our home and office regardless of location.

A report published in a research paper, State of IoT – Spring 2022, indicates that the number of IoT devices is expected to be connected globally and is projected to grow.

Aside from the practical uses of IoT devices for us humans, it also helps our societies around the globe by building smart retail and smart city solutions.

An IoT system consists of interconnected computing devices, mechanical and digital devices, objects, animals, or people that are provided with individual identification numbers and the capability to send and receive data over the Internet. Examples of this system include the Internet of Vehicles (IoV), the Internet of Buildings (IoB), and the Internet of Agriculture (IoA).

Aside from the practical uses of IoT devices for us humans, it also helps our societies around the globe by building smart retail and smart cities solutions.

Wikipedia.org defines IoT Devices as the following:

The Internet of Things (IoT) extends Internet connectivity into physical devices and everyday objects. Embedded with electronics, Internet connectivity, and other forms of hardware (such as sensors), these devices can communicate and interact with others over the Internet. They can be remotely monitored and controlled.

Wikipedia, https://en.wikipedia.org/wiki/Internet_of_things

IoT makes virtually everything “smart” and improves aspects of our life through the power of data collection, AI algorithms, networks, human interaction, etc. IoT technologies can also help monitor the health of a person with diabetes. It can also assist pet owners by using IoT GPS-tracking sensors to watch their pets in case of emergency, etc.

The devices that implement IoT operations collect telemetry of data which can be stored and hosted in cloud storage services like Azure. Big data, reporting, and analytics tools are also helpful in visualizing these collected data.

Making Sense of the IoT Technology We Have Today

The innovation of IoT helps us connect and exchange data with the sensors of these objects. It promises and envisions a digitalized future where devices can communicate to assist and enhance our daily routines.

Figure 8-1 shows us the different components of the IoT, which is essential to understand as an overview of its workflow and data processing before it gets to the user.

Figure 8-1. The different components of IoT: Sensors, Connectivity, Data Processing, and User Interface

Intelligent devices have sensors embedded into these IoT-compatible devices capable of sensing their environment—for example, IoT devices with sensors to check the current temperature and humidity indoors or outdoors.

These IoT devices, which include applications, smoke detectors, mobile phones, robot vacuum cleaners, etc., store this information from their sensors in the form of telemetry data with different properties, units, values, etc.

The sensors embedded in these devices continuously transmit data and connect to an IoT platform. The IoT platform then gathers all the valuable data collected by these IoT devices, which is processed for helpful information and then shows these data to the user through a user interface.

To manage and collect the data from these smart devices, the IoT platform includes its management interface for its users, cloud servers, databases, storage, etc. It processes the collected data by integrating and processing the information.

Furthermore, the platform analyses the data thoroughly to gather essential details and then sends back instructions based on the data provided. This data cycle is not limited to the one IoT device transmitting data back and forth to the IoT platform. The data is aggregated and shared with other IoT devices to improve performance and user experience. We need to consider the privacy of using these IoT devices.

We will go through the security concerns and challenges in IoT technologies later in this chapter.

Components of an IoT Solution

An IoT solution are composed of several components. _Table 8-1 shows us the different parts of an IoT solution that are useful to learn, especially if you are considering developing an IoT solution.

Table 8-1. An overview of the components of IoT

Component
Description

Sensors & IoT Devices

Smart devices with IoT sensors help us gather valuable data, which can vary in type and complexity. There are different sensors for temperature monitoring, smoke detection, motion detection, suspected behavior sensor on some security cameras, etc.

Connectivity

Connectivity means the data collected are synced from the IoT device sensors to the cloud for further processing. The typical communication transmission mediums include satellite networks, mobile networks, Bluetooth, WI-FI, WAN, etc.

Data Processing

When the data is collected, it gets to the cloud platform for data processing. However, data processing from IoT devices can be very complex, like detecting and trying to identify objects using computer vision on a video surveillance camera.

User Interface

When the data is collected, it gets to the cloud platform for data processing. However, data processing from IoT devices can be very complex, like detecting and trying to identify objects using computer vision on a video surveillance camera.

These applications can be developed to be event-driven, which means that they can be configured so that the user receives notices, critical alarms, etc., when a significant event or parameter happens. For example, an intelligent video security camera can send SMS, email or trigger a call with an alarm if an intruder and suspicion of robbery are detected while the user is away from home. The user can also perform some interaction with the IoT device depending on the type of application it is and its complexity.

So, as you noticed, the components of an IoT solution are more comprehensive than just sensors and devices. The IoT solution also requires connectivity, data processing, and a user interface which is easier to implement in a cloud-based solution.

Different Types of IoT Applications

The result and benefits of IoT solutions are being used widely in many organizations and industries today, including for home use and personal use.

Table 8-2 gives us a list of useful applications of IoT solutions that we have today and most of these IoT solutions are hosted in the cloud.

Table 8-2. The Different Types IoT Applications

Application
Description

Smart Thermostat Systems

It helps you save on power usage bills by knowing your usage patterns for heating during the winter and air conditioners during the summertime.

Activity Trackers

Activity trackers usually help us capture our heart rate pattern, calorie expenditure, activity levels, skin temperature, etc. Connected Cars

Connected Cars

IoT helps many car companies handle bill management systems, create features that allow drivers in their parking, update maps, driver profiles, and improve safety features.

Smart Electronic Plugs

These allow you to track a device’s energy level and directly get custom notifications and useful electricity analytics to your smartphone

Smart Home

By using our smart home devices with our mobile phones or smartwatches, we can take control and automate the schedules of IoT devices in our homes. For example, connected smart homes can help homeowners schedule their smart lights on schedule or based on sunset or sunrise. They can also get notified in real-time in case of a possible emergency while away from the premises of your home. For example, if the smoke detector alarm is triggered by smoke, it notifies and warns its homeowners.

Based on the number of IoT applications and how practical they are in our lives, we expect that the advancement in this technology will continue to evolve along with cloud computing development with edge computing, artificial intelligence, and machine learning.

Challenges of IoT

The IoT technology, its benefits, and its promising use cases generate opportunities for many enterprises, industries, and organizations to use devices or things smarter, automate them and use the IoT telemetry data to create tools and solutions that are beneficial to its users. Let us take a look at some reasons why IoT is practical and why it can also be a challenge.

The Advantages of IoT

Every technology that gained its popularity because people have seen its purpose and use usually evolves and gets developed. Let us go through some of the common vital advantages of making use of IoT solutions:

Technology Innovation

The benefits of IoT technology help us by learning more about it and collaborating to make them better. For example, with IoT, a manufacturer can use car sensors to collect data to analyze and improve the design and features of a car to make them more useful and safer for its users.

Customer Satisfaction and Engagement

IoT technology helps improve customer satisfaction by detecting problems and improving the processes, speed, and efficiency of communication supporting customers.

Reduced Waste and Sustainability of Resources

IoT offers real-time information leading to effective decision-making & management of waste at the industry level. Using IoT to detect possible ways to improve waste management effectively helps aid in achieving the goal of sustainability. For example, if a manufacturer sees an issue in several car engines, he can track those engines’ manufacturing plans and solve this issue with the manufacturing belt.

Real-Time Data Collection and Streams

The traditional method of data collecting has its drawbacks, namely that it is intended for inactive use. The Internet of Things makes it easier to take quick action on data. The conventional approach of collecting data has drawbacks but makes it easier to take rapid action based on the facts.

IoT in Industrial Farms

An excellent example of how IoT works on an industrial scale is smart farming. Farmers can now understand and grow their crops much better, which helps reduces costs and money. Smart farms usually have devices that connect to the Internet and provide real-time data about how the condition of the crops. The information about the crops can contain information such as the moisture balance, texture of the soil, levels of pesticides in the environment, wellness of the livestock, etc. Many of these techniques are currently used in farms that generate vast amounts of produce. Many small industry farms have yet to be introduced to IoT technology. One of the possible challenges is the cost of investing in IoT devices with sensors useful for farming and watering.

Employee Monitoring system

These days, many huge organizations and industries are integrating their existing systems with IoT technologies to provide profitable and efficient results. IoT is incorporated into the business sector through employee monitoring systems. Facial recognition trackers, fingerprint scanners, and iris scanners are some of the innovative IoT devices that help organizations monitor the attendance of their employees. Some companies install these devices at their premises entrances to track employee attendance and for security.

IoT Warehouse

Some modern warehouses use IoT machinery and devices to keep track of their product generation. With the increased demand for consumer goods, production is growing significantly. Using IoT in inventory management reduces manual work time and helps save money.

The Disadvantages of IoT

The list of advantages and benefits of IoT in different sectors might be trending and famous; however, the technology of IoT and its solutions still need to be fully developed.

The following are some of the disadvantages and essential things to be mindful of, regardless of whether you are building IoT solutions in your organization or using it as a consumer.

Security

The IoT technology gives us the capability of a modern ecosystem of connected smart devices; however, the system can be prone to security risks if the IoT platform and devices are not configured with sufficient security authentication, network, and endpoint protection.

Privacy

The use of IoT exposes a substantial amount of personal data, in extreme detail, without the user’s active participation. The fact that some personal data is being collected creates many privacy issues.

Flexibility

There is considerable concern regarding the flexibility of an IoT system. This concerns integrations between systems, as many diverse systems are involved in the process.

Complexity

The design of the IoT system is also quite complicated. Moreover, its deployment and maintenance are also not very easy. The design of the IoT system is also quite complicated for several reasons. First, to design and build an IoT system, you need to identify the business needs. Create the strategy plan, investigate what kind of IoT devices or types of sensors you need, learn how to configure them with your business operations, etc. Moreover, its deployment and maintenance are only effortless with good IoT management platforms or tools.

Compliance

Compliance is a recurring challenge for organizations using IoT solutions. Two primary issues make compliance with IoT challenging: lack of visibility into what’s happening with the devices being deployed in the field and an absence of well-defined IoT compliance standards.

So we have learned both the advantages and disadvantages of IoT and what we can do about it.

In the white paper "“Why IoT projects fail”, published by Beecham Research in 2020, they reported and estimated that nearly three-quarters of IoT projects wouldn’t be successful. Based on the numbers and research presented, a few top reasons are that the business needs to be better planned, technological problems that need to be foreseen, etc.

Explore IoT Security and Compliance Learning Resources

To learn further about the general cybersecurity standards of IoT solutions, the IoT Security Compliance Framework will help guide you through how you can make your IoT solutions secure and compliant.

Microsoft’s Well-Architected Framework for IoT Workloads is suitable for learning more about the best practices of security and compliance considerations in IoT.

On the technological front, one known challenge is the inability to link the IoT telemetry data and business processes effectively. Another challenge is establishing technology standards and dealing with data privacy, compliance, and security risks.

To tackle these challenges, it is crucial to have a solid organizational strategy for implementing IoT solutions. Additionally, it is essential to understand the possible technical challenges that arise beforehand.

IoT in Microsoft Azure

One of the reasons Azure is recognized for their IoT platform is due to their cloud services. These can be pre-customized or fully customizable solutions, providing options for companies early on in their IoT strategies and more advanced in their implementation.

Cloud platforms can help quickly scale IoT systems to include IoT-supported devices from different manufacturers. They also provide support for advanced data analytics and machine learning capabilities.

Azure IoT

IoT refers to a collection of a managed platform of services across the cloud and edge computing. The Azure IoT helps users to connect, monitor, and manage billions of IoT devices from different locations, etc.

Azure IoT platform is built with security, performance, data management, and analytics that will aid organizations, businesses, and their users in performing what they need to do to deploy and manage their IoT applications for their end-users.

The IoT has different components that are necessary to make things work together:

Things

Things are usually physical devices, objects, or things connected to the cloud persistently or intermittently. Examples of these things are industrial equipment or appliances with sensors.

Insights

The information collected by the physical IoT device or things is called Insights. They will be processed, analyzed, and turned into actionable knowledge by people working with them or using AI.

Actions

Actions are the verbal term used when people take action on the data. For example, if a user does something about the insights provided through the IoT platform and then uses them for their business operations, existing applications, or tools.

These three components are essential in an Azure IoT because they can be designed and built using different cloud technologies based on your use case.

There is an open public website called AppSource, which publishes a list of applications built by IoT solutions.

IoT Hub and its Device Provisioning Service

The IoT Hub is a central hub where you can manage and control the communication between your IoT devices and their applications. Nearly any device can be embedded with sensors, and software to capture data can be added via an Internet connection.

If you are working with IoT development in Azure, IoT Hub and its IoT device provisioning service might be something worth considering. Otherwise, other cloud providers also have their corresponding IoT hub for your use cases as well.

Furthermore, when you connect any IoT device in the cloud as an IoT service, you can collect helpful telemetry and data, like temperature, humidity, etc., and its sensors. These data telemetry will be sent in real-time to a central place where you can manage the data and create solutions. For example, integrating it with event-driven and serverless solutions within Azure, etc.

IoT Central

Azure IoT Central is a secure IoT application platform that scales with your business demand increases. This Azure service for IoT solutions helps organizations by removing the hassle of creating IoT solutions from scratch. This platform helps in the management of IoT at the enterprise level in-house.

If you have the requirement to connect and manage IoT devices up to millions of them across the globe, using Azure IoT Central is ideal to use. It enables you to provision your machines, visualize the data telemetry collected from devices, and monitor them with the capability of integrating other valuable services for alerts and communication.

Developers have the capability and option to develop with the IoT Central REST API. They can create this API and integrate it with new or existing applications. Using the REST API for IoT application development, you can provision your devices and manage their operations.

IoT Plug and Play App

Using the Azure IoT services, you can connect virtually to any IoT device or a simula‐ tor using your mobile phone. The IoT Plug and Play is an app for mobile systems like Android and iPhone that can turn your phone into an IoT device. This is especially handy if you want to get started or still need an IoT device but are still waiting to invest a lot of money. You can quickly explore the IoT features without configuring a dedicated device using the IoT Plug and Play.

To start using the app, you need your smartphone as an IoT device instead of an IoT device. This is ideal if you are learning IoT development in the cloud.

Furthermore, the app sends the telemetry data from your smartphone sensors and allows you to send commands to it from the cloud-based IoT service on Azure. Check out this guide on Microsoft to get started using your smartphone as an IoT device.

IoT Dev Kit and Azure-Accredited IoT Devices

The IoT Dev Kit (AZ3166) is an all-in-one IoT development kit designed and optimized for users or developers who want to create prototypes and develop IoT solutions leveraging services in Azure and other third-party services.

Figure 8-2 shows the different sensors included in the kit. Essentially, it’s a smart IoT device compatible with Arduino, an open-source prototyping platform for building digital devices.

Figure 8-2. An illustration of the IoT Dev Kit sensors

Its user interface board includes features such as an OLED display for data, different sensors, a hardware debugging chip (ST-Link), a security chip, etc. There are sample IoT projects available with documentation that will help you get started with this device.

This IoT Dev Kit can also use IDE coding tools like Visual Studio Code with Arduino Extension to quickly build a full-fledged IoT application. The kit integrates with multiple services like IoT Hub, Logic App, and Cognitive Services.

Its rich user interface board includes features such as an OLED display for displaying data, different sensors, a hardware debugging chip (ST-Link), a security chip, etc. There are available sample IoT projects in a digital catalog with documentation that will help you get started with this device.

Digital Twins

The Digital Twins is an IoT platform that will help you create and design a digital view of real-world things such as places, people, business processes, etc. This platform for IoT also helps you gain insights that are useful in helping you and your organization deliver better products, optimize operations and costs, and create breakthrough customer experiences.

Digital Twins is considered a platform as a service (PaaS) that will help you develop twin graphs based on digital models of entire territories, including buildings, industrial factories, farms, energy networks, stadiums, railways, stadiums, etc. Since the models created are digital, you can gain insights that will help optimize business operations, help reduce costs, and help provide satisfactory customer experiences.

You can develop a digital twin that reveals the IoT devices in the broader cloud solution. This can be connected to IoT Hub device twins to send and receive live data and telemetry.

Device Twins vs. Digital Twins

Although it is easy to confuse these two, the IoT Hub device twins differ from Azure Digital Twins. Although the IoT hub supports IoT Hub device twins for each IoT device you connect to, digital twins in Azure can be anything defined by digital models and instantiated within Digital Twins.

Usually, IoT device twins are composed of JSON documents which typically contain information about the state of a device. Information that may be stored includes device configuration, metadata, etc.

Additionally, a few examples of the difference between IoT Hub device twins and digital twins by looking at the following JSON code files.

Example 8-1. A code example for Device Twin in JSON format

{
  "deviceId": "example-device-twin",
  "modelId": "dtmi:com:example:TemperatureController;1",
  "version": 15,
  "properties": {
    "desired": {
      "thermostat1": {
        "__t": "c",
        "targetTemperature": 22.8
      },
      "$metadata": {...},
      "$version": 4
    },
    "reported": {
      "serialNumber": "deviceserialnumber77899",
      "thermostat1": {
        "maxTempSinceLastReboot": 25.3,
        "__t": "c",
        "targetTemperature": {
          "value": 21.8,
          "ac": 200,
          "ad": "Successfully executed patch",
        }
      },
      "$metadata": {...},
      "$version": 11
    }
  }
}

Example 8-1 is a code snippet that illustrates how a device twin is represented in JSON format. The JSON file is composed of properties like the id of the device twin, serial number, and other helpful metadata.

Example 8-2. A code example for Azure Digital Twin in JSON format

{
  "$dtId": "example-digital-twin",
  "serialNumber": "deviceserialnumber77899",
  "thermostat1": {
    "maxTempSinceLastReboot": 25.3,
    "targetTemperature": 21.8,
    "$metadata": {
      "targetTemperature": {
        "desiredValue": 21.8,
        "desiredVersion": 4,
        "ackVersion": 4,
        "ackCode": 200,
        "ackDescription": "Successfully executed patch",
        "lastUpdateTime": "20220-11-17T06:11:04.9309159Z"
      },
      "maxTempSinceLastReboot": {
         "lastUpdateTime": "2022-11-17T06:10:31.9609233Z"
      }
    }
  },
  "$metadata": {
    "$model": "dtmi:com:example:TemperatureController;1",
    "serialNumber": {
      "lastUpdateTime": "2022-11-17T06:10:31.9609233Z"
    }
  }
}

Example 8-2 is a code snippet example that illustrates how a device twin is represented in JSON format. If you compare closely by focusing on the property key serialNumber, you notice that the JSON file is composed of properties almost like the Example 8-1 except this property serialNumber is defined differently.

I recommend the article: Manage IoT Plug and Play digital twins and Digital Twins API documentation to learn more about how you can practically use these examples.

When implementing and using Digital Twins, you’re typically charged based on consumption, and pricing is based on the following parameters: Query Units, Operations, and Messages.

An IoT developer tool, Azure Digital Twins Explorer, can work with your Azure Digital Twins instances data, such as twin graphs and models. These data can be used to interact and visualize this digital twins explorer.

Securing IoT on Azure using Defender for IoT

In the earlier section of this chapter, we discussed security and compliance considerations as a few challenges of IoT technology. If your IoT solutions are hosted on Azure, Defender for IoT helps secure your IoT solutions.

Defender for IoT helps organizations automatically discover and detect unsecured and unmanaged assets in their IoT devices or solutions. Additionally, the built-in security features, such as security posture management, endpoint threat detection, and IoT Hub integration, help protect your IoT services and solutions in the cloud.

It helps by detecting critical vulnerabilities, security threats, and possible real-time anomalies. It provides quick and informative insights leveraging analytics and machine learning for suspicious behaviors. These insights are helpful in a way that helps minimize the need for manual setup.

Furthermore, Defender for IoT also includes network sensors on-premises to capture the traffic from or to devices. This application is then connected to a port or a tap, network analyzers, a tool used to check what is going on with network communication using packets, etc.¹.

Network analyzers can play a crucial role in protecting IoT devices and their associated networks against various security threats and attacks.

These analyzers may monitor and examine the traffic on a network for signals of malicious behavior, such as infection by malware, attempted network intrusions, and other security issues. This is useful for ensuring the safety of IoT devices. As a result of conducting this network monitoring, businesses can determine the origin of the assault, the nature of the attack, and the particular IoT devices that have been compromised.

Additionally, network analyzers can be used to assess the execution of IoT networks, such as the reliability and velocity of data transmission, and identify any bottlenecks or areas for improvement. They help optimize the performance of IoT networks and make them more secure and resilient against potential security threats.

Another use case for network analyzers in IoT security is to monitor and analyze the wireless signals emitted by IoT devices, such as Wi-Fi or Bluetooth signals, for signs of unauthorized access or interference. Typical IoT attacks can be avoided using these analyzers.

Overall, network analyzers are essential tools in defending IoT networks against security threats and attacks, providing critical information and insight that help improve the security and performance of these networks.

Map Services

Now that we have learned about IoT in Azure, let us check what Map Services are for and what they are capable of.

Map Services provides a collection of cloud-based mapping services that help with geospatial assistance. The SDKs of Map Services help develop map solutions. This SDK is useful for mapping data that is useful in creating geographic features for applications for the web or mobile. These services include components that help manage data types like route, geospatial, map search, rendering, and other essential components.

Some of the features of Azure Maps include the following:

Geospatial Services

A set of REST APIs allows developers to perform geospatial operations such as geocoding, reverse geocoding, routing, and traffic data.

Interactive Maps

A web-based map control allows developers to quickly add interactive maps to their web applications.

Location APIs

It is a set of REST APIs that provide access to data related to points of interest, geofencing, and other location-based information.

Spatial Operations

A set of REST APIs that allow developers to perform complex spatial operations such as geofencing, buffering, and spatial analysis.

Real-Time Traffic

A set of REST APIs that provide real-time traffic information, including incidents, congestion, and traffic flow.

Mobility Services

A set of APIs that provide information related to mobility, such as public transit information and electric vehicle charging locations.

If you want to develop solutions with Azure Maps, a free account for this service is available to help you get started. It also has web and mobile development SDKs, which allow developers to add map services to their new or existing applications.

Limitations of Unsupported Regions of Azure Maps

It is essential to take note that Azure Maps services are available in most countries and regions except parts of China and South Korea. Therefore, when you are hosting or deploying your applications for Azure Maps, you need to ensure that you are not using the Azure region you choose to be supported.

Maps Power BI Visual supports the visualization of spatial map data, which helps visualize business data based on the location context or categories.

In addition, Azure Maps can also be implemented and developed with low-code/no-code tools like in Power Apps. You can create an interactive map with canvas apps.

Controlling Results of Azure Maps Service using Geographical Scope

Azure Maps has localization support depending on which country or region you are implementing its solutions. It is a maps service that is supported globally; this allows you to specify a geographic scope. By being able to determine your geographical area, you can limit data residency due to compliance regulations. For example, select the data residency to the European (EU) or the United States (US) geographic areas.

All requests are typically stored in the specified geographic area. A cloud provider like Microsoft also supports replicating customer data to other Azure regions within the exact geographic location to fulfill high availability and disaster recovery.

For example, if you are using Azure Map’s European API as your geographical end point, your maps requests or data resides in a European Azure data center. This API allows end-users or consumers to choose where their map data can be accessed.

Azure Maps European API provides high-quality, reliable, and accurate data for mapping and location-based services in Europe. You can integrate it into different applications, which are highly scalable and protected, making it a recommended choice for organizations looking to build mapping-based applications in Europe.

If you are in Azure Geographic areas like Europe, your API geographical endpoint would be eu.atlas.microsoft.com. If you are in the United States region, your ideal API geographical endpoint would be us.atlas.microsoft.com. Additionally, when using the Azure Government Cloud, use the atlas.azure.us endpoint.

Azure Maps provides a set of REST APIs for various geospatial operations, including geocoding, routing, and traffic data. These APIs are available globally and can be used to build applications in any country.

The API provides data for mapping and location-based services for countries worldwide. The scope of the Maps API varies by country, so it is critical to review the detailed documentation for the country you are interested in for more information about the general data and services.

Authentication and Security on Azure Maps

The support for Azure Maps for authentication and security is important. There are several mechanisms for authentication and security that it provides.

Authentication via Azure Active Directory (AAD)

The AAD is the default authentication mechanism for Azure services. The AAD authentication support in Maps allows you to use your Azure-synced credentials service.

API Key Authentication

Authentication using the API keys is the simplest way to authenticate requests to the Azure Maps API. You can generate API keys in the Azure portal and then pass the key as a query parameter in your API requests.

OAuth 2.0 Authentication

Since Maps support OAuth 2.0 authentication, you can gain secure access to the API on behalf of your users. If you intend to utilize the API to gain access to data that is restricted to a certain user or if you want to use Azure Maps as part of a bigger application, this can be handy. It can also be beneficial if you need to utilize Azure Maps as a component of a larger application.

In addition to the authentication options highlighted, it also has the following security features to safeguard data and applications in Azure.

Secured Encryption via TLS/SSL encryption

Using this option as a security option, all API requests and responses are encrypted using TLS/SSL to protect sensitive information while transiting over the Internet.

Network Security Groups (NSG)

NSGs are used widely in networking in Azure. Implementing NSGs allows you to control the inbound and outbound traffic to and from your Azure Maps account. You can define rules that allow or block the traffic based on destination source, IP addresses, ports, and protocols.

Role-based Access Control (RBAC)

RBAC allows you to control access to Azure Maps by defining roles and permissions for users. It is possible to assign roles to users and groups and then specify what actions they can perform with the API.

Finally, Azure Maps generally provides potent security elements to protect your applications and data. It is a cloud service that supports high scalability and reliability, essential in designing and developing these applications with map integration.

Maps Integrations with Azure Event Grid

The Azure Maps service integrates well with Event Grid if you want to handle events and notifications. It is a fully managed event routing service that allows users to send event notifications to other Azure services and can even trigger processes. It is also a cloud service that supports event streaming of Azure resources.

Figure 8-3. An example of Maps integration with the Event Grid service

The Publish-Subscribe model of Event Grid can also be programmed and developed with serverless event-driven services in Azure. Figure 8-3 illustrates how Azure Maps can be integrated using Webhooks as the trigger with the Event Grid service.

Webhooks are valuable if you want to integrate Azure Maps and Event Grid with external services or APIs. As one of the supported features of Event Grid, webhooks allow you to subscribe to specific events and have Event Grid deliver those events to an HTTP endpoint in real-time.

When an event occurs, it can send an HTTP POST request to the specified webhook URL containing the event details. The recipient can then process the event and perform proper measures. This provides an easy and adaptable way to integrate Azure services and automate workflows.

For example, you could create a webhook subscription for an Azure Blob Storage container so that every time a new blob is created, Event Grid sends an event to your webhook, which in turn could trigger a series of actions such as copying the file to another location, processing the data, or sending a notification.

Overall, webhooks in Event Grid enable you to build event-driven applications and automate workflows across Azure services easily.

Developing with Azure Map Services

It is easy to develop or improve your applications with Azure Maps services. Let’s walk through several ways to do that.

Develop Using REST APIs for Maps Search Service

One of the ways to develop Azure Maps service is by using the Web SDK. Azure Maps Search Service is a RESTful API designed to help web developers search addresses, places, and business listings by name, category, and other geographic data.

Azure Maps Search Service is a geocoding and search service that allows you to search for addresses, places, and points of interest (POIs) using a straightforward REST API. The service provides information such as geocoordinates, addresses, and place names for power mapping and location-based applications. You can use the API to retrieve geo coordinates, addresses, and location names for energy mapping and location-based applications.

To use the Maps Service REST API, you must have an Azure Maps subscription and obtain a key. Once you have these prerequisites, you can create the API URL, send an HTTP GET request, and parse the JSON response to extract the relevant data.

The Maps Search Service includes multiple search features:

Request coordinates for an address (geocode address location) using the Search Address API.

Using the Fuzzy Search API, you can also search for an address or Point of Interest (POI)

You may perform a Reverse Address Search to translate the coordinates to understandable street address

Translate coordinate location into a into a user-friendly format using Search Address Reverse Cross Street API

Do a Reverse Address Search to translate the coordinate location to the street address.

Before developing with REST APIs, a prerequisite is required. You need to set up an Azure Maps account and have a primary subscription key to use the Azure Maps Search Service and its APIs.

Developing Using Web and Mobile Software Development Kits (SDKs)

Suppose you need to develop Azure Maps services other than REST APIs. In that case, there is also support for Web and Mobile SDKs for different languages and applications developed with Azure.

The following SDKs are available for Azure Maps services.

Maps Web SDKs

The Azure Maps Web SDK can be used with the Map Control client-side JavaScript library. This library enables developers to utilize the functionality and features of embedded Azure Maps and render maps into their websites or applications, regardless if they are for mobile or web.

Maps iOS SDKs

Maps iOS SDKs are software development kits (SDKs) for the iOS platform that allow mobile or cross-platform developers to easily integrate mapping and location-based functionality into their iOS apps. The iOS SDKs provide tools and libraries that simplify accessing and using Azure’s mapping and location services. With minimal effort, developers can add powerful mapping and location functionality to create applications that provide a good and engaging user experience.

Maps Android SDKs

Maps Android SDK is a vector map library for the Android platform. With this SDK, developers can add powerful mapping and location functionality to their Android applications with the least effort, allowing them to build applications that deliver a rich and engaging user experience.

Finally, Azure Maps is designed to be easy to use and integrate into various applications, including web applications, mobile apps, and IoT devices. Additionally, it is highly scalable, secure, and globally available, making it a reliable choice for organizations looking to build mapping-based applications.

Learn By Doing (Try it out!)

The following quickstart tutorials are recommended as they are updated based on Microsoft’s technical updates for the service. This section is recommended but optional.

Microsoft Learn Hands-on: Deploy a pre-built module to the Edge device

Quickstart: Connect Azure IoT Dev Kit to Azure IoT Central

Tutorial: Connect an IoT Plug and Play module (C#)

Azure IoT Development Kit

Quickstart: Send telemetry from a device to Azure IoT Central

Microsoft Learn Hands-on: Azure IoT Developer

Quickstart: Create a map in an Android app

Quickstart: Create a map in a web app

Summary

In this chapter, we learned the fundamental concepts of IoT and how it gives us practical solutions and innovative technological capabilities.

We also learned about Azure’s different IoT solutions and cloud technologies that make these IoT devices easier to design, develop, and manage.

You also learned about the Azure Maps Services that enable us to build and integrate geographical map solutions with our web and mobile applications.

Finally, I covered the known advantages, disadvantages, and challenges that IoT solutions might encounter, how we can handle these challenges, and how to get started building with these IoT solutions.

Check Your Knowledge

After learning about the IoT concepts in this chapter, what are these technology’s most valuable benefits to us and society?

What are the top three challenges in IoT innovation that we need to solve as developers, engineers, or technologists?

Why do you think building IoT solutions on a cloud platform is valuable and efficient?

To find the answers to the Check Your Knowledge, turn to the back of the book in the Appendix

Recommended Learning Resources

Azure Percept overview

Azure Percept sample AI models

TechAhead: Evolution of Internet of Things (IoT): Past, present and future

Microsoft Defender for IoT Ninja Training

Azure Maps Sample Gallery

Microsoft: Create an interactive search map with Azure Maps

University of Oxford: Artificial Intelligence: Cloud and Edge Implementations

Microsoft: IoT Central REST API Reference

Microsoft Defender for IoT Ninja Training

¹ Extrahop.com, “What Is Network Traffic Analysis (NTA)?” by Chase Snyder, February 2022, https://www.extrahop.com/company/blog/2018/what-is-network-traffic-analysis-nta/

Part IV. Cloud Integration Services, Cloud Infrastructure, Cloud Development and DevOps in Microsoft Azure

This fourth section of the book and its chapters focus on teaching the foundational concepts of artificial intelligence, machine learning, Internet of Things (IoT), Edge technologies, security, and identity management in Microsoft Azure.

Chapter 10 focuses on Microsoft Azure integration services. In this chapter you will about the the different cloud technologies that you can integrate with your cloud-native applications, incuding the existing ones (on the cloud or on-premises). This chapter covers Azure API Management, Azure Logic Apps, Azure Service Bus, Azure Web PubSub, Azure Event Grid, Azure Logic Apps, etc.

Chapter 11 guides on how to develop systems or applications with modern DevOps tools we havev today. You will learn about what DevOps, Azure DevOps for team collaboration for developers and IT operations. You will also learn about automating DevOps pipelines, version source control, Github Actions etc. How to monitor and troubleshoot Azure resources using Application Insights. You will also get to know about other cloud technology services like Azure DevTestLabs, App Configuration, Azure Bicep, ARM Templates and more. Know what Infrastructure as a Code (IaC) is and how it helps with automation of deployments.

By the the end of this section of Section 4 you alreayd have gained the important technical skills and fundamentals you need to know in building modern applications in Azure. You also would be able to identity which cloud integration technologies that are best to use based on your use cases.

I hope that you enjoy learning this section.

Chapter 9. Azure Cloud Integration Services and Tools

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 10th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at jleonard@oreilly.com.

Modern applications are distributed in the cloud and on-premises, and integrating applications and data is inevitable.

- Chezhiyan, Lead Product Consultant at Kovai and Serverless360

Up to this point in the book, you’ve learned about cloud computing and the different services in Azure. The focus has been computing, networking, analyzing, securing, and developing solutions.

In this chapter, we shift gears, and I will cover Azure’s different integration tools and cloud services. These integration services will give you the fundamental knowledge and understanding to prepare you for integrating and connecting your applications and services hosted in the cloud. You will also learn how to connect on-premises applications with cloud services using the integration services in Microsoft Azure.

In this chapter, we shift gears, and I will cover Azure’s different integration tools and cloud services. These integration services will give you the fundamental knowledge and understanding to prepare you for integrating and connecting your applications and services hosted in the cloud. You will also learn how to connect on-premises applications with cloud services using the integration services in Microsoft Azure.

Cloud Integration - An Introduction

Cloud integration links cloud-based applications, networks, and services to establish a seamless and unified environment. It enables sharing of data and applications across multiple platforms and improves businesses’ effectiveness, responsiveness, and scalability.

Integration methods on the cloud connect different cloud systems and applications using middleware technologies and APIs. It ensures that their components can communicate and share data. One example is integrating cloud-based apps such as CRM (Customer Relationship Management) and ERP (Enterprise Resource Planning) systems.

Cloud integration can facilitate the streamlining of business operations, the reduction of manual processes, and the enhancement of data precision. It can also improve team collaboration and data sharing and facilitate improved decision-making. Organizations and businesses benefit from improving their operations by integrating systems and applications in the cloud. Modern cloud integration provides reliable, scalable, and faster data synchronization between different systems, regardless of where they are hosted - on the cloud or on-premises.

Types of Cloud Integration in Azure

There are multiple cloud integration methods, and each varies depending on its purposes, such as point-to-point integration, replication of data, and API-based integration. Internal IT teams or external vendors specializing in cloud integration can perform it. Common integration can be performed with cloud computing technologies, different environments, and methods. Let us review the common scenarios and uses where cloud integration is applicable.

Application-to-Application Integration

In this type of integration, various applications or services are integrated within the organization to improve functionality. This application-to-application integration can be on-premises, cloud, or hybrid. Integration involves issuing requests and commands to trigger business events or processes.

Data Integration

Integrating data means synchronizing data in different resources. These data are sourced from databases or other data resources and can be transported or processed during the data integration.

Cloud-to-Cloud Integration

Cloud-to-Cloud integration connects the process of combining two or more cloud-based applications or services to facilitate data sharing and shared functionality. Integration between cloud environments is possible via APIs, multi-cloud technology solutions, and other middleware technologies.

On-Premises to Cloud Integration

On-premises-to-cloud integration involves linking legacy or on-premises systems to cloud-based applications or services. Middleware technologies, such as cloud connectors and VPNs (Virtual Private Networks), can accomplish this.

Hybrid Cloud Integration

Integrating on-premises systems and cloud-based applications or services to create a hybrid environment is the objective of hybrid cloud integration. Integrating hybrid environments with intermediary technologies such as iPaaS and hybrid integration platforms is possible.

Cloud-to-Mobile

This integration involves linking apps that use the cloud or services with mobile phones and tablets to improve data sharing and functionality. Integration between the cloud and mobile applications is possible via mobile application development frameworks and API-based integration.

Cloud-to-Business Integration

Cloud-to-business integration connects cloud-based apps or services with business partners or customers to facilitate data sharing and functionality. The cloud to enterprise systems integration can be achieved through various means, such as EDI (Electronic Data Interchange) and web services.

Overall, each type of cloud integration has its requirements and differences. They can also be combined. Aside from the kinds of integrations mentioned, there are additional methods of enabling applications and systems to communicate and share data. These include external APIs, web services, SaaS (Software as a Service) solutions, B2B (Business to Business), Internet of Things (IoT), and event-driven and microservices technologies that allow distributed applications to get integrated to build cloud applications.

Benefits of Cloud Integration

Every organization has challenges and problems that integration solutions can solve. For example, most organizations have to handle time-intensive tasks such as inputting data into two or more systems. This duplication of effort is frequently manual and uses needed resources that could be better served on other charges. On the more severe side, organizations also face challenges such as unexpected system failures or data loss.

These problems and challenges can affect the productivity and efficiency of the organization’s workers, and if the systems are in production, they may also affect the business users and customers. This can lead to unhappy users, customers, business partners, or unproductive teams.

Let us overview how well-integrated systems can help with the IT challenges mentioned.

Reliability and Scalability of Applications

There’s much to say regarding reliability. I’ll cover the topic briefly in this book, but if you’d like to understand better, I recommend “Designing Data-Intensive Applications” by Martin Kleppmann. He discusses the importance of reliability in our systems. Also, he covers other essential topics, such as bugs in business applications that can affect productivity and result in unexpected expenses or damage a company’s reputation.

IT systems are often built with the ambitious goal of being reliable and scalable. A reliable system should handle unexpected disasters or significant outages, affecting not just an organization’s employees but the entire business.

Reliability is an application’s ability to operate consistently and predictably under different conditions and over time. A reliable application should minimize downtime, errors, and data loss and recover quickly from failures or disruptions.

You need the right technologies, tools, and knowledgeable engineering teams to build a reliable system to plan and implement a good cloud integration strategy. These teams must have the organization’s business purposes in mind and collaborate well with other teams to ensure that things work across the organization.

Scalability is closely related to reliability because a scalable application must be designed to handle the current and future workloads as the application grows. A scalable application that is not reliable can cause significant problems for its users and the business, such as lost revenue, damaged reputation, or legal liability.

You can think of scalability as the capability of an application to accommodate increasing levels of demand, usage, or growth without degrading its efficiency, availability, or reliability. Scalability is crucial because as an application acquires more users or data, its burden and complexity increase, which can cause it to become sluggish, crash, or unreliable.

A highly scalable application can smoothly adjust to these shifts by adding resources, enhancing its architecture, or streamlining the algorithms it uses to preserve its stability and performance.

Furthermore, other benefits such as backup options for data, high-level security to protect your data, monitoring, disaster recovery, and other observability tools are also part of increasing the reliability of systems. Most of the cloud providers we have today, including Azure, provide integration tools such as Azure Monitor, Application Insights, and other built-in tools within each Azure resource; however, integrating existing systems to any cloud platform takes time and effort. If planned well and done correctly, data and systems integrations in the cloud can significantly benefit any organization. In general, developing a scalable and dependable application can ensure that it can adapt to the changing needs of your users and the market, all while delivering a positive user experience and safeguarding your business’s interests. To accomplish reliability and scalability, you must consider a good application architecture, robust cloud infrastructure, reliable cloud services, automation, monitoring tools, and adopting best practices for security, performance, and resilience.

Improved Work Efficiency and Cost Savings

Manual operational tasks in an organization can be time-consuming. Cloud integration eliminates repetitive activities, streamlines workflows, and enhances efficiency by removing the requirement for switching between various apps. This helps save time, diminishes mistakes, reduces labor expenses, and eliminates the need for infrastructure and maintenance on-site. Cloud integration allows businesses to spare money, remain competitive, and adapt to shifting business and market requirements.

With cloud integration automation, tasks can be done by routines managed by a service. This eliminates user error and reduces the time spent on tasks. As a result, teams can focus on more complex tasks, thus making better use of their time. This reduction of person work hours is a cost-effective strategy.

Business Agility and Better Business Processes

Integration with the cloud enables businesses to link and combine several cloud-based apps, services, or data sources to produce a unified source of reality for information. It allows organizations to collect, analyze, and exploit data in real-time from various sources, offering them immediate understanding that can help them optimize their processes, uncover new possibilities, and make decisions based on accurate information.

An approach driven by data is significant because it enables organizations to make decisions using the data rather than making assumptions or using their intuition. This results in increased performance, profitability, and competitive advantage.

An exciting research, “Cloud Computing for Agility, Complexity, and Speed” by Christina Delimitrou, highlighted how cloud computing management could be improved using data-driven approaches. She said, “Don’t rely on people to solve this problem. Rely on the data. Rely on the system to tell you how to optimize the system.”

The data-driven approaches with cloud computing can be made possible with the help of cloud integration tools available. Cloud integration and agility in business can be a good synergy and produce good results.

Cloud Integration on Azure

Azure provides modern solutions for connecting and integrating services, APIs, and applications that can help your business and engineering teams. Azure has services for API management and other integration tools in different deployment models that give new opportunities for developing business models. When there are opportunities to create new business models, there are also opportunities to increase business revenue.

Furthermore, Azure also provides solutions that can help secure and produce your applications and your enterprise data without compromising performance, scalability, reliability, and so on.

In the following sections of this chapter, you will learn about the fundamentals of Web APIs, and the different and valuable Azure integration services that will help you design, build, and develop better systems, services, and applications in the cloud or hybrid scenarios.

Introduction to Web APIs

APIs in software engineering or cloud development usually refer to Web APIs accessed over the Internet, like our favorite websites or web applications. They are components that act as mediators between applications or systems that allow them to communicate with each other. This allows us to have handy things like an app on your smartphone that gives you up-to-date information about the day’s weather. These APIs connect data, apps, and web servers to enrich experiences. Figure 10-1 illustrates this relationship between APIs, our smart devices, and data held in the cloud.

Figure 9-1. APIs connect us to data, apps, and web servers.

Different Types of Web APIs

To understand how cloud integration in Azure works, learning about the different types of Web APIs is also good. The Public APIs, Partner APIs, Internal APIs, and Composite APIs are specific Web APIs with other purposes. Let us review the different purposes of these types.

Public APIs

These are APIs exposed to third-party developers and others in the community. Typically, these APIs enable developers to create applications that can access the services or data of a company. Frequently, public APIs are used to expose social networking platforms, weather information, and travel reserving services.

Partner APIs

Similar to public APIs, partner APIs are intended for use by a specific group of associates or customers. These APIs are frequently used to integrate multiple software systems or to facilitate specific business processes. Typically, partner APIs are more reliable than public APIs because authorization and access controls are required.

Internal APIs

Internal APIs share data and services between teams and departments within a corporation or organization. These APIs are not generally available to people or developers from third parties and are predominantly used to enhance internal workflows and communication. Internal APIs are frequently employed to automate business processes, reduce development time, and improve team collaboration.

Composite APIs

The Composite APIs types combine data or services from multiple sources into a single API. This allows developers to access multiple services with a single API query and various services with a single API query, thereby reducing complexity and enhancing performance. They are implemented in enterprise applications that require data access from multiple systems or applications.

In conclusion, Public APIs are accessible to the general public and externally facing. Partner APIs resemble public APIs but are designed for a particular group of partners. Composite APIs aggregate data or services from multiple sources into a single API. Internal APIs are used internally within an organization.

API Management Lifecycle

An API has a lifecycle requiring some structure and control management. A good API management solution should support the entire lifecycle of an API, as shown in Figure 10-2, which typically includes planning and initial design, development, implementation, testing, deployment, operations, versioning, and retirement.

Figure 9-2. API management lifecycle.

The API lifecycle management benefits businesses or organizations in different ways, including the ability to create, design, develop and manage APIs for their applications. It enables teams to adopt the API-first strategy and maximize productivity. API lifecycle management uses proven methods to design, build, and deploy APIs. These methods reduce confusion and unnecessary work, improving teamwork and production.

Furthermore, it helps creates a roadmap for every API-related project, giving executives more API visibility. It provides the basis for a successful API monitoring plan for monitoring every API’s health, efficiency, and consumption.

As its teams develop each API, its lifecycle will differ depending on the business use case. The following stages can help organizations and their teams in standardizing their processes, policies, and workflows for APIs developed with cloud applications.

Plan

Developers and API architects plan and create the API, considering its use, functionality, and performance. They should also create a shared workspace and a GitHub repository with a CI pipeline. These stages stabilize the API lifespan and set stage-specific locations and tools. Generally, the planning stage of this lifecycle helps in preparing and improving API quality beforehand.

Design

API design needs thoughtful data presentation choices. Users and other programs can comprehend an API definition’s intended functionality. OpenAPI and AsyncAPI define APIs. These specs standardize API definitions, contracts, documentation, mocks, and testing. API definitions must comply.

Develop

After designing the API, developers build code to implement the functionality. Most development teams utilize Git. They can track changes and securely revert. They use GitHub or GitLab to store their source code, track issues, and review work. To standardize their approach, the team must identify software development workflows, which vary widely.

Test

APIs are tested during development and Deployment. Developers and QA teams can verify API functionality. CI/CD pipelines or several locations can automate or manually test APIs. Early testing helps couples identify and fix issues before they reach production.

Secure

The API lifecycle checks an API for common vulnerabilities that could compromise an application’s security. 6. Proof This stage documents the API and its documentation. It’s crucial to check that an API’s authentication logic confines data access and API interaction to authorized users. CI/CD pipelines can manually or automatically perform API security checks to ensure that all an organization’s APIs meet security standards. Access control, encryption, and IAM can safeguard the API.

Deploy

Deployment is releasing APIs across environments. Many teams use CI/CD pipelines and API gateways to streamline Deployment and test and secure updates before releasing them to customers. Agile teams that release new code many times a week need consistent methods to make deployments more predictable.

Observability

In production, API telemetry data is gathered, displayed, and notified. SREs and DevOps engineers will automatically configure monitors to inform them of API performance and security issues. They’ll also employ contextual API performance data APM tools. API observability helps identify API errors, delays, and security vulnerabilities before they damage dependent services, partners, and customers.

Retirement

APIs might become obsolete. The API is now being deactivated and archived.

Although the stages are defined, every API lifecycle is different and unique. The most important factor to remember is that teams that follow a well-defined API lifetime tend to be more productive and can produce a high-quality API. A practical API governance approach requires a stable API lifetime, establishing its framework.

Azure API Management

Azure API Management (APIM) is a flexible integration service in Microsoft Azure that allows users to design, develop, and implement solutions to connect applications, APIs, and different systems. It is a Platform as a Service (PaaS) platform for APIs of different environments. It supports the entire API lifecycle and hybrid scenarios, even for multi-cloud use cases.

In terms of pricing options, APIM has different pricing tiers depending on use cases, implementation types with other Azure services, or if it is for non-production or non-production environments:

Consumption - For lightweight, implement with serverless solutions, and if you prefer to pay per use basis

Developer - Typically used for non-production or development uses and does not support SLA

Basic - Ideal for a small or entry-level production use

Standard - Ideal for medium or traditional production use

Premium - Used for large enterprise applications or systems that have high demands for using API management

Isolated ¹ - Similar to the Premium tier for enterprise use cases for production use

Each APIM tier has a different set of features and capabilities. Check Microsoft’s Documentation for the updated support per pricing tiers and limitations.

Benefits of Azure API Management

One of the common problems that APIM helps its users or consumers is solving security risks of exposing APIs hosted in Azure or externally to the public.

The following are some additional benefits of Azure API Management:

Enables and supports adopting the complete API life cycle for different use cases in the cloud, hybrid, or even in multi-cloud environments

Provides capabilities for a good user experience for integration with IoT applications, web or mobile apps, smart watches, wearables, etc.

Business-to-business (B2B) integration capabilities that give options to users to communicate and exchange business data

Creates opportunities for collaboration and removes the overhead of point-to-point integration

Supports compliance and security management of APIs, such as configuring authorization token swaps, certificates, etc.

Helps in the abstraction and modernization of the backends of legacy applications by enabling them to be accessible by modern cloud apps or APIs

Provides support for API security using Azure networking services like Azure Private Link

Allows flexibility of developing APIs with microservices architecture

Best Practices of Desiging and Implementing Web APIs

If you want to dive deep into the technical details of RESTful web API design for cloud applications, check Microsoft’s guide for best practices and recommendations for implementing APIs in your applications.

Besides API management’s benefits, it enables cloud integration’s functional capabilities by supporting other Azure resources and services, and it also has observability features within Azure’s platform

Table 10-1 lists and describes the Azure services compatible with Azure API Management which includes the most common and useful tools for both monitoring, security, IAM, development, networking and many other good services to integrate APIM with.

Table 9-1. Azure Resources that are compatible with Azure API Management

Azure Active Directory (Azure AD)

Developer authentication and authentication of API requests.

Azure Monitor

This resource is commonly used to monitor logs, set up alerts, and perform API management operations and events.

Azure Key Vault

Holds secrets, client certifications, and password management for your API management.

Application Insights

It is part of Azure Monitor but specifically helps get valuable insights for troubleshooting using end-to-end tracing, live metrics, etc., of Azure API Management.

Azure Functions

Azure Functions is a serverless compute service that allows you to develop event-driven serverless solutions in your applications and APIs provisioned in Azure API management.

Azure Web Apps

This resource is typically used for hosting applications in the PaaS deployment model. It provides options and features of integrations with Azure API Management.

Azure Logic Apps

Low-code or no-code tools for automated workflows and orchestrations compliment API management when communicating with your APIs, client applications, other backend services, and SaaS platforms.

Azure Service Fabric

Pricing tiers like Developer and Premium support direct integration of Azure Service Fabric with API Management. Using API management with Service Fabric, you can send traffic to a stateful or stateless web service or backend API.

Azure Networking

API Management can be integrated and configured with Azure networking services like setting it for network-level protection using Application Gateway, private endpoint, virtual networks, etc.

Azure Event Hubs

Event streaming tasks can be integrated using Azure Event Hubs with Azure API Management to associate an event stream with your API requests.

APIM is a flexible integration service in Azure that allows users to design, develop, and implement solutions to connect applications, APIs, and different systems whether in the cloud, hybrid, or multi-cloud environments. It supports organizations and developers to manage the entire API lifecycle.

Components of Azure API Management

Aside from learning about the benefits of APIM in general, let us learn about the crucial elements and key term concepts you need to know. APIM has three major components: the API Gateway, the API Management plane, and the Developer Portal, as shown in Figure 10-3.

APIM has three major components: the API Gateway, the API Management plane, and the Developer Portal, as shown in Figure 10-3. learning-microsoft-azure\images\chapter10\api-management-components-handdiagram.jpg

Figure 9-3. Azure API Management service components

Let’s go through these three components to understand better what they are used for and how each can help you design your APIs with APIM.

Azure API Gateway

When implementing APIM service to build and integrate your APIs for your applications, you must first set up the API gateway. The API gateway is the first point of contact for all requests from your client application.

Once the client requests come via the API gateway, it will then forward those specific requests to the backend APIs. When this communication happens, API providers can integrate and communicate with an API’s implementations and backend architecture without affecting the consumers.

The API gateway supports security, routing configuration, caching, throttling, quotas, rate limits, etc. It also allows you to perform security verification of API keys and credentials, such as certificates or JWT tokens, during API operations or requests.

You also have the option to configure your API gateway to improve response latency and improve workloads of backend services by caching the responses.

API Management Plane

The API management plane allows API providers to comprehensively manage and interact with their APIs. This component can be used to provision and configure the API management service. Furthermore, users can also set up other settings and capabilities, such as packing the APIs as products, managing the service users, and, most of all, defining APIs. Suppose you set up OpenAPI specifications and other backend services with your APIs. In that case, you can import the schemas and manage monitoring, networking, and analytics using the API management plane.

Tools such as the Azure Portal, Azure CLI, Azure Powershell, and supported SDKs for common programming languages, you can use the API management plane to import the schemas and manage monitoring, networking, and analytics.

Developer Portal

If you work with application development in the cloud in Microsoft Azure, you must work with integration services and development tools to use APIs. The Azure API Management service provides a portal within Azure Portal for developers to design, build, and test their APIs with their client or backend applications.

The Developer Portal is an open-source website that allows you to customize it according to your API and your company’s branding or consumer content preferences. You can use the portal to call or interact with an API using the console. You can also create accounts, manage the API keys, read the API documentation, set up personalized analytics, download the API definitions, etc.

These three major API management components make integrating APIs, web services, and applications easier to design, develop, implement, and manage on Azure’s platform.

Technical Details of APIM Components

If you want to dive deep into these components’ technical details, I recommend reading Microsoft’s official documentation.

Azure Logic Apps

In Chapter 3, you learned about Azure Functions for creating serverless workflows programmatically by code. Azure Logic Apps also help enable you to author workflows in a low-code way, just like low-code or no-code platforms in Microsoft.

Azure Logic Apps provides many integration capabilities and workflow features to help you and your organization design, orchestrate and automate your business processes. Azure Logic Apps enable users with no programming experience to create automated integrations and workflows with the Azure Logic Apps’ visual designer.

The web-based workflow visual designer allows you to create integration workflows and orchestration for your data, systems, and applications. Furthermore, you can also configure it for Business-to-Business(B2B) and Enterprise Application Integration (EAI).

Benefits and Uses for Azure Logic Apps

Azure Logic Apps offers low-code connectivity, a wide selection of connectors, a stable and flexible platform, and monitoring and management. These features ease integration procedures and automate corporate processes, enhancing productivity.

The following are some of the possible use cases you might want to consider for using Azure Logic Apps for integrations with your existing or new applications:

Schedule or create workflows for sending emails when someone uploads a file to an Azure storage or new data saved into a database

Automate some business processes from on-premises applications to the cloud by using Logic Apps to help with order management processing

Monitor events in your IoT solution by setting up alerts by email, Slack, Teams, and other communication channels for communication or notifications

Event tracking and monitoring using other Microsoft SaaS services like Azure

B2B Integration features that work with other services like Azure Service Bus, Biztalk Server, Azure Functions, Azure API Management, etc.

Logic apps as shown in, Figure 10-4 integrates well with Azure AD and APIM to route events and data to backend systems which can be other resources like an app service, an Azure Function app, or a SaaS service like Salesforce, Dropbox, and other supported SaaS services. Developers can work with their application development through APIM’s Developer Portal with authentication and IAM through Azure AD.

Figure 9-4. A simple example of enterprise integration in Microsoft Azure using Azure Logic Apps with other Azure services like Azure API Management, Azure AD, and backend systems (On cloud and on-premises)

Pricing Tiers of Azure Logic Apps

Choosing an appropriate Azure Logic Apps pricing tier has significance because it impacts cost efficiency, scalability, integration capabilities, security, and compliance. It also matters to avoid spending too much, to guarantee scalability, and to obtain the necessary features for integration, safety, and compliance. Assessing your business’s requirements before selecting a suitable price tier is vital.

Azure has three pricing tiers for creating Azure Logic Apps depending on your use case scenarios:

Consumption – This tier allows users to pay for what is being executed or used. This tier supports hosting multi-tenant and Integration Service Environments (ISE).

Standard – This tier also has the Pay-As-Go pricing model. It supports single tenants, such as App Service Environment (ASE), etc.

Premium – This tier has a fixed pricing mode known as Service Bus Premium Messaging. It is ideal for production use cases with mission-critical systems with requirements for high throughput and the ability to configure scalability based on the workloads required and deliver predictable performance.

Logic Apps has a variety of various pricing tiers, each of which varies in terms of both price and the services it includes. If you select the appropriate pricing tier, you will only have to pay for the functionality that you require, which will help you save money.

Integration Service Environment (ISE) Retirement in 2024 and Migration

In August 2024, the ISE will retire because of its dependency on the classic Azure Cloud Services. If you are currently implementing ISE with Azure Logic Apps in existing applications, please read this recommended https://learn.microsoft.com/en-us/azure/cloud-services-extended-support/in-place-migration-overview [migration guide] by Microsoft.

The Azure Pricing Calculator is a great cost estimation and management tool for implementing Azure Logic Apps. This tool is also helpful for considerations of switching tiers, plans, etc., as the business requirements demand it. Microsoft also has a good guide for planning your costs for Logic Apps.

Azure Logic Apps Components

Azure Logic Apps have four key components: workflow, trigger, actions, and built-in connectors. They are essential elements that compose the service. These components allow for integration with various systems and applications, automation of business processes, visual depiction of workflows, and a low-code strategy for integration. Let’s go through each one.

Workflow

A workflow is a sequence of steps describing the flow of a business process, task, or workload you want to implement. Workflows are visual representations of the automated business processes in Logic Apps. Workflows are essential because they depict the automation process clearly, making it easy for users to understand, alter, and sustain their organization’s operations.

Trigger

Every workflow must always begin with a single trigger when using a logic app. A trigger will be activated when a specific condition is satisfied, such as when a particular event occurs or data satisfies specific requirements. Numerous triggers have scheduling features that allow you to control the frequency with which your workflow is executed. Following the activation of the trigger are any number of steps that begin running and executing. These actions either handle the process, convert the data moving through the workflow, or move the workflow to the subsequent stage.

The Logic Apps workflow designer, accessed through the Azure Portal, Visual Studio Code, or Visual Studio, allows for the graphical creation of workflows. The fundamental specification of each workflow is written in JavaScript Object Notation (JSON), and this definition is included with each workflow. Altering this JSON specification allows you to edit workflows if that is what you would instead do. Furthermore, support for Azure PowerShell and Azure CLI commands is provided for certain operations relating to creating and managing Azure Logic Apps.

Action

An action is a task or job that needs to be executed during an Azure Logic App workflow. An example of a step in a workflow can be sending an email through built-in connectors like Outlook or Office 365. Using this workflow step in Azure Logic Apps, you can set your logical condition, like the specific recipients to send alerts or notifications, etc.

Built-in Connectors

The built-in connections that run natively in the Azure Logic Apps allow users to control the structure and manage the logic of the workflow. Logic Apps enables developers to optimize business processes and incorporate services and systems by creating workflows. The built-in connectors allow developers to connect and integrate with various systems swiftly, and services are one of the primary advantages of this Azure service.

These connectors provide predefined integrations that allow connecting to various services and systems. Such connectors are used with standard services such as Salesforce, Dropbox, and Twitter. The low-code/no-code features enable users to work efficiently without creating backend code and APIs. They provide a basic and straightforward method for developers to access data and initiate actions within these services.

Among the primary advantages of using built-in connectors in Azure Logic Apps are:

Developers are not required to invest time developing custom code or APIs to connect with different applications. This help saves time and effort during development.

Built-in connectors include pre-built actions and triggers, making it simple for developers to integrate with different services. The integrated connectors are available for various services, allowing developers to construct flexible workflows.

These connectors are also designed with security, ensuring data is safely transmitted and stored.

Built-in connectors are an integral component of Logic Apps and make it simpler for developers to integrate with different services, automate workflows, and create effective business processes

Some built-in Logic Apps connectors are not explicitly associated with any Azure or Microsoft service, and customization is flexible based on the use case. It is worth checking out the different built connectors available per pricing tier. Now that you have learned the concepts of Logic Apps and its everyday use cases.

In the next section, you will learn about Service Bus, another excellent integration service in Azure that works well with API Management, Logic Apps, and other Azure services.

Azure Service Bus - Cloud Messaging Broker Service

One of the common reasons why enterprises and organizations consider using cloud computing is for data integration between applications. Cloud computing enables integration through enterprise messaging services, which helps implement messaging to transfer data between different types of applications, APIs, or web services. A messaging service uses a message to transfer data between applications, regardless of whether they are hosted in the cloud, on-premises, or hybrid. The data transmitted in a messaging service broker in the cloud can be in different formats like plain text, XML, JSON, etc.

Service Bus is a cloud enterprise messaging service in Azure that serves as an intermediary broker that applications and services can use to transfer data or information from one application to another. It has benefits and capabilities that help solve complex messaging problems that many developers have to solve programmatically regarding passing data, messages, or information between distributed enterprise applications.

Data integration between decoupled applications needs messaging that is reliable and secure. Service Bus guarantees secured, reliable, and scalable message delivery between applications through Azure’s platform. it is possible to send or receive asynchronous messages or data from one to many systems using queues, topics, etc., in the Service Bus service.

In addition, you can use one-to-many relationships messaging delivery using the Topics and Subscriptions. The comparison of these will be discussed further in the coming sections.

The following are common uses of Azure Service Bus:

Serves as an intermediary broker that applications and services can communicate to transfer data or information from one application to another. It can also be integrated with other resources within Azure, like serverless integration with web applications, IoT solutions, Event Grid, Azure Logic Apps, etc.

Delivers messages for multiple applications. It supports several parallel competing consumers, meaning multiple applications can read from a queue. A topic can also be used for the publish-subscriber pattern, which will be discussed further in the later section of this book.

Provides reliable messaging for decoupled and distributed systems. Some applications are designed to be decoupled and distributed for many reasons. A few known reasons are to improve flexibility, scalability, and reliability. Azure Service Bus provides reliable messaging capabilities between multiple distributed or decoupled systems.

Acts as a message broker in the cloud capable of handling transactions (operations grouped to be executed together) with atomicity. It also guarantees that the integrity of its message storages in its internal operations. An example of its features is that this capability supports a dead-letter queue (DLQ) in case of message failure. It is essential to note that transactions are only supported in the Premium or Standard pricing tiers.

Supports one-to-many communication. Many business processes today require the implementation of communication of one-to-many-through publisher-subscribers 1:n relationship. Azure Service Bus Topics and Subscriptions support these kinds of use case scenarios.

Aside from the everyday use cases listed, Service Bus has other advanced features. These features include auto-forwarding, which allows you to forward or chain messages between topics or queues in the same namespace, duplicate detection, prefetch, message deferral, etc.

Furthermore, Service Bus supports common programming languages for cloud development. For example, developers or cloud engineers can design, develop, and integrate systems using C#, Visual Basic, F#, Java, Go, C/C++, PHP, Ruby, Java Messaging Service (JMS), and other supported languages.

Support for Java Messaging Service

Azure Service Bus Premium Tier supports Java Message Service (JMS) 2.0 API. JMS is a standard service used in accessing middleware services for messaging in Java programming language. You can integrate and communicate with Azure Service Bus from your Java applications using the Advanced Message Queueing Protocol (AMQP 1.0) protocol.

You can fully integrate the Azure Service Bus with other Azure services, such as Azure Event Grid, to respond to any events coming through Azure Event Grid and Azure Logic Apps. It also supports serverless development. For example, you can use it with Azure Functions using the Azure Service Bus bindings and its triggers for events happening in a specific resource or application.

Combining Azure Service Bus and Azure Functions allows you to develop a robust messaging and event-based architecture that enables real-time communication and processing between multiple apps and services.

Listed below are some examples of how these can be combined and integrated. * Process Messages Between Applications Using Queues: Azure Functions can be used to ingest messages from a Service Bus queue, process them, and perform the necessary actions, such as modifying a table in a database, delivering an email, or triggering another Azure Function.

Trigger an Azure Function Based on a Queue Message: Azure Service Bus can send queue messages to an Azure Function using the Service Bus Trigger of Azure Functions. You can process the message and execute the necessary duties through this serverless integration between these Azure services. It is useful when you need to initiate an action based on a specific event, such as the arrival of a new customer order or the completion of a lengthy task.

Implementation of Publish-Subscribe Pattern: Service Bus gives developers a publish-subscribe technique that can be used to facilitate immediate communication between services and applications. Subscribe to topics or queues and handle messages as they arrive using Azure Functions.

Combining the features of Azure Service Bus and Azure Functions, you can develop a highly scalable, event-driven architecture that enables real-time communication and processing between cloud-based services and applications.

You can also add Service Bus in your low-code or no-code automated workflows using Azure Logic Apps to manage your messages or data. You can use Azure Service Bus with Power Platform’s Dataverse in Dynamics 365 trigger to create a decoupled architecture where events in Dataverse trigger actions in other services and applications through Service Bus and Power Platform. The integration with Azure Stream Analytics for queues and topics for processing outputs is also supported.

Depending on the use case, you can use Azure Service to transfer data or information immediately, by schedule, or with delay, using enterprise message broker systems or services. If you are using .NET, you are most likely already using Azure Service Bus; however, Apache ActiveMQ is another type of message broker commonly used to develop solutions using Java for enterprise applications.

Azure Service Bus and Apache ActiveMQ are both message brokers used these days. Both work with message providers like JMS to send and receive messages between client applications. They have similar capabilities, like queues and publish-subscribe semantics.

On the other hand, they have some differences too that are worth checking out and considering before implementation.Deep diving into the details of technical differences is beyond the scope of this book.

Please read Microsoft’s guide about migrating JMS applications from Apache ActiveMQ to Service Bus..

Azure Service Bus Components

The Service Bus delivers a reliable and expandable messaging infrastructure that can be used to construct distributed apps and services. They provide for secure and private communication between applications, enable several kinds of messaging patterns, and interface with other services offered by Azure to allow real-time information processing and analysis. Moreover, they support a wide range of messaging patterns.

Figure 9-5. Azure Service Bus Queues (multi-tier apps)

Let’s go through the components that comprise the core of Service Bus’ messaging capabilities: Namespaces, _Queues, Topics and Subscriptions.

Namespaces

An Azure Service Bus Namespace is required before creating a queue, topics, or subscriptions for message processing or delivery in Azure Service Bus service. A namespace is like a hub or the logical scoping container of your Azure Service Bus messaging services like the topics, queues, etc.

Figure 10-5 shows how data flows between the namespace and distributed systems. The advantage of the Service Bus namespace is that you can gather all your Azure Service Bus queues, issues, etc., for typical distributed applications in one place.

Queues

By design, distributed applications are usually not intended to communicate directly with only one application. It means these applications can communicate and be integrated even if not in the same hosting or instance. A messaging broker, such as the Azure Service Bus, sends messages or data. When a distributed application needs to send a message to another application, it must use queue storage. In Azure Service Bus, a queue acts as the logical container of messages an application sends that can be stored, retrieved, processed, or retrieved from another application or web service.

Azure Service Bus queue uses the algorithm concept of the FIFO (First In, First out). When an application’s message arrives at the queue, it is held in this cloud messaging queue storage until it is time to be retrieved in the order that they arrive. This messaging service provides advanced features that help you manage and handle the messages between distributed applications depending on each unique use case.

In Figure 10-5, you will find an illustration of how Service Bus works in different multi-tier applications. Distributed applications, like a web app, mobile app, or web service, can pass and send data to another application using Azure Service Bus Queue service.

Topics and Subscriptions

The Azure Service Bus topics, like queues, act as cloud messaging brokers. The difference is that the subject uses one-way communication, meaning it uses the subscription to deliver messages to its consumers or receivers, also called subscribers.

Publisher-Subscriber is one of the cloud messaging architectural design patterns for distributed applications in the cloud. This design pattern is supported in Azure Service Bus service with topics and subscriptions, where users can simultaneously send messages or events to interested consumers.

Topics and Subscriptions is a feature that developers can use for establishing a publish-subscribe messaging pattern. This capability enables an asynchronous, decoupled communication model between multiple components or applications.

Let us review how it each of them works and operates:

A topic is a logical entity representing a message stream. For example, when a message from an application is published to a Service Bus Topic, it is sent to all associated subscriptions and subscribers.

A subscription is a named view on a topic that specifies a subset of the topic’s communications. Subscriptions allow applications and components to filter and receive only the messages that are relevant to them.

Using Topics and Subscriptions, developers can construct a messaging system in which publishers and subscribers are unaware of one another. A publisher can send messages to a topic and then to all interested subscriptions. Subscribers can filter messages based on various criteria, such as message properties or content, and receive only the relevant notifications.

This feature is useful when multiple components of an application or multiple applications need to communicate with one another but are not tightly coupled or dependent on one another. Examples of practical uses in which Topics and Subscriptions may be utilized include:

A news publishing platform in which different news categories are published on various topics, and subscribers can choose to receive only the news that interests them.

A stock trading application in which users subscribe to the equities they are interested in and only receive updates for those stocks.

An IoT application in which various devices publish data on distinct topics, and various applications or components can subscribe to these topics to receive the data that is pertinent to them.

Overall, Topics and Subscriptions in Service Bus provide developers with a robust messaging functionality that promotes a decoupled, asynchronous communication model between components of an application or between applications.

Figure 9-6. Azure Service Bus Queues transfer application data or messages between web applications.

Figure 10-6 illustrates how Service Bus Topics are used with subscriptions to deliver messages to different applications. When there is a new message published to the Topic, its consumer applications or subscribers will receive the message.

Code Examples for Azure Service Bus

Code examples for sending and receiving queue messages or subscribing to Azure Service Bus Queues, Topics, and Subscriptions, are available in the book’s repository on GitHub.

Choosing the Right Azure Cloud Messaging Implementation

Azure Service Bus is beneficial in brokered messaging for decoupled applications, enabling better data transfer, communication integration, and other helpful cloud messaging features. Choosing the right cloud messaging implementations can take time and effort. For example, when selecting between Azure Service Bus queue or Azure Storage, you should assess how your use case fits each service’s features.

You typically use Azure Service Bus over Azure Storage Queue if you need a FIFO guarantee of your queue messages. Also, you would use Azure Service Bus to group your messages into transactions or if the use case requires you to consume a batch of multiple messages for your distributed applications.

On the other hand, Azure Storage is ideal over Azure Service Bus queues if, for example, your use case needs a standard queue without any required additional features. Consider Azure Storage Queue over the Service Bus Queue if your queue message exceeds 80GB.

The Service Bus Queue can only handle messages between 64KB to 265 KB if you use the Standard pricing tier, or if you have the Premium tier, you can send up to 100 MB.

In the next section, you will learn about Azure Web PubSub, another sound cloud messaging service in Azure that is ideal for building web apps with real-time features.

Azure Web PubSub

Azure Web PubSub is a managed service within Azure that enables developers to create scalable real-time messaging applications. Web PubSub enables developers to transmit and receive messages in real time, allowing them to develop applications that require instant updates, such as chat applications and live dashboards.

Web PubSub provides numerous advantages, such as simple integration with other Azure services, built-in security features, and the capacity to manage millions of connections and messages. Web PubSub is typically used for chat applications, real-time monitoring, and live transmission.

There are three pricing tiers for Web PubSub: free, standard, and premium (public preview).

Free: The Free tier supports up to 20 connections, 20,000 messages per day, Max Units of 1, and an SLA of 99.9%, which makes it ideal for testing and development.

Standard: The Standard tier offers increased performance and scalability, with up to 1,000 connections, unlimited messages per day (the first 1 million messages are complimentary), and Max Units of 100 with an SLA of 99.9%. It also incorporates message broadcasting and message ordering capabilities. The Standard plan is billed according to the number of connections, messages, and data transferred.

Premium: Currently, in Public Preview, the Premium tier offers the same features as the Standard Tier with additional features such as enhanced message reliability, autoscaling, customized domain name, and Availability Zone support. The Premium tier is meant for applications that require the most significant degree of reliability and performance for enterprise production workloads.

In the previous section, you learned about Azure Service Bus, for reliable messaging service for sending and receiving messages in the cloud between distributed applications. These messages can then be delayed or processed based on preferences and system design.

The Web PubSub, on the other hand, is a PaaS service in Azure that enable its users to subscribe or send-receive messages between client and web services in real-time using WebSockets. As a persistent web communication protocol, a WebSocket uses a TCP connection between a web client and a web server for bi-directional real-time communication.

Figure 10-7 shows that WebSockets allows live contact between the web client and server.

As illustrated in the figure, Azure Web PubSub uses the following sequence: . The web client sends an initial HTTP request to the web server . The web server confirms and accepts the request from the client . When a client-server handshake is created, the client and the server can communicate and exchange messages in real time.

Figure 9-7. Azure Web PubSub web communication via WebSockets.

WebSockets are preferred over HTTP connections because of their speed. Websockets are bidirectional, while HTTP connections are unidirectional. HTTP communication takes longer because data must be requested before the web server can send data or messages back to the client.

Azure Web PubSub service uses serverless and native WebSockets, enabling users to fully manage it on Azure’s cloud platform and develop it with their applications. It supports publish-subscribe messaging and uses cases for web applications and mobile apps requiring real-time data transfers or communication.

The Architecture Pattern Used in Azure Web PubSub

Like the pattern used in Topics and Subscriptions of Azure Service Bus, the Publisher-Subscriber pattern (also called pub/sub messaging) is used on Azure Web PubSub. It allows us to build modern applications that relay or send messages to many subscribers asynchronously, as shown in Figure 10-8, where the publisher sends the message to an input channel. The input channel sends them out to its interested subscribers to an output channel through a message broker.

Figure 9-8. Publisher-Subscriber pattern is used for communication and sending/receiving messages from one application to its multiple interested subscribers.

The publisher-subscriber steps are as follows:

The application that acts as a sender (Publisher) sends a packet of data events in the form of messages. These messages sent will be transmitted to the Input Channel.

The input channel communicates with a Message Broker

The message broker then distributes these messages received to an output channel

The output channel then sends the message to its consumers (Subscribers).

Using the Publisher-Subscriber pattern, distributed and cloud-based applications can send messages from/to different systems components. This pattern helps avoid interruptions. It can assist in blocking the Publishers from waiting for any response from any subscriber since they are decoupled and isolated from the Subscribers or receivers.

The Publisher-Subscriber pattern also provides advantages such as the assurance of the reliability of sending messages through its support for asynchronous messaging and the ability to handle failures efficiently through its use of message brokers in the cloud. In addition, this type of messaging pattern separates concerns for distributed applications.

This pattern promises some good features regarding its asynchronous messaging capabilities for decoupled applications and systems. Consider using this pattern to build applications that send information to multiple consumers. This pattern is ideal for your applications to perform data processing well by adopting eventual consistency.

Benefits of Azure Web PubSub

Web PubSublets you quickly develop real-time messaging web apps by utilizing WebSocket, both native and serverless. This real-time functionality enables publishing content updates between the server and connected clients, like an app for mobile devices or a web application with a single page.

The following are some uses for Azure Web PubSub:

Real-time messaging between clients and web servers

Live data dashboards

Reliable and high-frequency updates of data, content, feeds, and reports, e.g., live auctions, gaming, voting polls, etc.

Multi-platform real-time collaboration, e.g., live pair programming workspace

Live chat and real-time communication, e.g., online customer service, web assistant, etc.

Internet of Things (IoT) real-time monitoring

Send real-time alerts, notifications, collaboration tools, etc.

Automating events

Build serverless applications with it using Azure Functions in your desired programming languages

Decoupled or distributed systems or microservices

In addition to these use cases, Azure Web PubSub also allows client-to-web server connections for large-scale applications with support for scalability, high availability, the ability to create many instances, client connections, etc.

Developing Azure Web PubSub is flexible and easy because of its broad support for different types of clients. For example, it supports web client apps for web, desktop, mobile, IoT, servers, game consoles, etc.

The Fundamentals of Azure Web PubSub

To develop with Azure Web PubSub, you must first understand some essential concepts, terms, and components. Let’s go through each of them.

Server

The Web PubSub service’s primary purpose is to handle the connections, events, messages, etc., communicated and transmitted from or to the web client application. The server in the Azure Web PubSub service can play the role of the server-side event listener and the event handler.

Message

A message is data in text format, JSON, or binary. When the client has established the connection to the server using the WebSocket connection, it can send messages to another application. The maximum message that can be sent is 1MB.

Hub

A hub is a logical concept for grouping places in your client connections based on purpose. For example, you may create a hub for client connections related to chat communications, notifications, or alerts. Each hub can be associated with an Azure Web PubSub service. You may associate different applications in unique hub names within the same service of Azure Web PubSub.

Group

Aside from the hub in Azure Web PubSub, you can also create a group. The client can join or leave the group anytime. In Azure Web PubSub, a group session can have multiple clients, and a client can join or connect to different group sessions simultaneously. In this case, if a message or notice is sent to a group, it will be received by clients connected to that group session.

Client Events

Any event initiated by the client is considered a client event. Examples of client events include a connection through WebSocket, sending or receiving messages, connection states, etc.

Event Handlers

An event handler in Azure Web PubSub is used to handle the logic of incoming events happening in the client. Event handlers manage the client events that should be filtered for further processing. To integrate the event handlers with the client events, it is necessary to register and set up the event handlers through Azure CLI or the Azure Portal beforehand.

Event Listeners

Event Listeners wait and listen to incoming client events. The Web PubSub service listeners cannot interrupt communication between the client, the service, and the server.

Users

Azure Web PubSub connections that are associated with a user. Each user can connect to multiple devices, browsers, or sessions.

Now that you have a handle on the essential concepts, you can learn about the ypical workflow of Azure Web PubSub that you will learn in the next section.

Typical Azure Web PubSub Workflow

A typical connection workflow for Azure Web PubSub, as shown in Figure 10-9, usually starts in the following sequence. . Client initiates the request to connect to the Service . The Service invokes the Server using the Cloud Events protocol, an event handler over HTTP . The Server gives a response to the Service through the REST API . The Service sends the message response from the Server to the Client

Azure Web PubSub workflow diagram image::images/chapter10/azurewebpubsubworkflow.png[]

Azure Web PubSub supports most of the common programming languages and a wide variety of client applications. After learning about Azure Web PubSub and its common uses, let us check how to get started developing with it.

Initially, you need to create the Azure Web PubSub service in Azure. There are different ways to build it. You make it directly on the Azure Portal, the Azure CLI, or deploy it as infrastructure as code using a Bicep template.

In the later chapters of this book, you will learn more about infrastructure as code technologies such as Azure Bicep and other cloud command line tools for Azure.

Furthermore, before you can create these workflow for Web PubSub in Azure, you need to create and setup your service. Figure 10-10 shows an example of creating the Azure Web PubSub service through the Azure Portal

Figure 9-9. A simple illustration of Azure Web PubSub creation in the Azure Portal

When selecting the resource name for the Web PubSub service, and it is vital to note that it needs a unique name.

Figure 9-10. Azure Portal – The different pricing tiers options for Azure Web PubSub creation.

Whether you use Azure Web PubSub for your own Dev/Test hands-on learning or actual production workloads, there are appropriate and flexible options. To create and provision Azure Web PubSub, you can choose the free, Standard, and Premium tiers, as illustrated in Figure 10-11.

Like in any Azure development, you must add this resource into an Azure Resource Group, as shown in the Azure CLI scripts in Example 10-1. After creating the Azure resource group and its region, you use the command az extension add --upgrade --name webpubsub to ensure you have the latest Azure Web PubSub extension version.

Example 9-1. Azure CLI - Create a Resource Group

//Create a resource group
az group create –name "<the name of your resource group here>" -location "EastUS"

// To create the Azure Web PubSub resource into your newly created resource group using Azure CLI

az webpubsub make --name "<your-unique-resource-name>"
--resource-group "<name of your resource group>"
--location "EastUS" --sku Free_F1

After creating the Azure Web PubSub resource using the AzureCLI commands, your hostname will be made in the following format: your-unique-resource-name.webpubsub.azure.com, which will then be used in the development of applications with the Azure Web PubSub.

Code Examples for Azure Web PubSub

A supplementary code example specific to developing with Azure Web PubSub is available in the book’s repository.

Azure Event Grid

If you want to create and build applications that support an event-based PubSub pattern, then Azure Event Grid is an excellent service to consider. It allows you to develop reactive applications to changes or events. You will typically use Azure Event Grid if you want to do some actions when a specific event occurs, such as sending an email if significant events are happening with an Azure Virtual Machine.

Figure 10-12 shows that Event Grid integrates well as the event communication layer between its event resources and the different types of event handlers, such as the workflows, message brokers, Azure Automation, and other supported handlers. This event-tracking service can be integrated well with other services in Azure or even third-party or on-premises services by routing events to its distributed subscribers or registered receivers.

You can integrate applications using events with Azure Event Grid, which is a highly elastic, serverless computing event broker. Event Grid delivers events to subscriber destinations such as apps, Azure services, and any endpoint with network access.

The following are some benefits to using Azure Event Grid for cloud integration development:

Event-driven notification and messaging delivery that supports HTTP-based technologies within Azure and support for external APIs or integration

Reactive programming and development that helps you build a better, more reliable applications, capitalizing on guaranteed event delivery and the cloud’s high availability.

Simplicity and flexibility for event-based applications. Event Grid can be used as a managed service for managing the distribution of events from any source to any destination, simplifying your event-based applications.

Minimize latency by eliminating polling because Event Grid decouples applications using a pub/sub pattern and simple HTTP-based event delivery, enabling the development of scalable serverless applications, microservices, and distributed systems

Components of Azure Event Grid image::images/chapter10/azure-event-grid-overview-diagram.jpg[]

Furthermore, building with Azure Event Grid for integrating applications to be reactive to certain events can benefit many. Organizations that want to extend their business processes and implementations can use this service by implementing custom event handles, customized events, etc.

Learn By Doing (Try it out!)

The following are some of the recommended QuickStarts from Microsoft’s official documentation. Hands-on exercises also relate to this chapter’s topic in this book’s code repository.

Import and Publish API on Azure API Management

Tutorial: Publish and subscribe messages using WebSocket API and Azure Web PubSub service SDK

https://learn.microsoft.com/en-us/azure/azure-web-pubsub/tutorial-serverless-notification?tabs=csharp?WT.mc_id=AZ-MVP-5004251 [Microsoft Quickstart Tutorial for Creating a Serverless Notification App with Azure Functions and Azure Web PubSub Service]

Microsoft Quickstart Tutorial: Create a chat app with Azure Web PubSub service

Microsoft Quickstart Tutorial: Monitor Virtual Machine Event using Azure Event Grid and Azure Logic App

Summary

In this chapter, you have learned how cloud integration technologies and services make it easier to collaborate and synergize the different business processes and systems flexibly. Applications that adopt modern cloud integration tools benefit from the scalability, reliability, cost-efficiency, and better business agility by getting more flexible opportunities to connect, transform and integrate their data and other business processes into other applications.

You learned about some of the cloud integration services and tools available in Azure, including Azure API Management, which offers comprehensive API management capabilities for consumers, developers, and API providers, and Azure Logic Apps, a low-code/no-code integration platform that enables users to create stateful serverless workflows with ease.

Additionally, the chapter covers Azure messaging services such as Azure Service Bus and Azure Web PubSub, which offer different ways of sending messages to distributed applications, web services, APIs, or systems.

Finally, the chapter highlights the benefits of using Azure Event Grid for event management with applications, allowing for better control and reactivity to changes and events, whether on Azure or on-premises.

Check Your Knowledge

Aside from the benefits of cloud integrations to applications on-premises, hybrid, or in the cloud, what are the other benefits?

How can Azure API Management service help applications with backends suitable for a microservices architecture?

Based on what you have learned about Azure Service Bus, what other helpful use cases or scenarios exist for brokered messaging technology?

What Azure integration service would you consider using if you were to build a collaboration dashboard where users can join, connect, and collaborate in real-time?

When would you consider Azure Event Grid vs. Azure Event Hub?

To find the answers to the Check Your Knowledge, turn to the back of the book in the Appendix

Recommended Learning Resources

Microsoft Documentation for API Management

Book: Mastering Azure API Management by Sven Malik

https://azure.microsoft.com/en-us/blog/microservices-an-application-revolution-powered-by-the-cloud/ [Microservices: An application revolution powered by the cloud]

Microsoft’s API Design e-book

Microsoft Azure API Management Documentation

Azure Service Bus – A Brief Technical Overview

https://learn.microsoft.com/en-us/azure/service-bus-messaging/ [Azure Service Bus – Microsoft Documentation]

Microsoft Azure Web PubSub Examples

¹ Preview at time of writing

About the Author

Jonah Andersson a.k.a. Jonah Carrio Andersson is a Filipina-Swedish, Microsoft Most Valuable Professional (MVP) for Azure, Microsoft Certified Trainer, Software Engineer, DevOps Engineer, Tech Mentor, International Public Speaker and an Inspiring Tech Community Leader based in Sweden

She has a background in BSc. Computer Science and Agile System Development in Microsoft .NET and Java. She is also Microsoft Certified Trainer(MCT) who teach Microsoft Certifications related to Azure. A Microsoft Certified Azure Developer and DevOps Engineer Expert, and other certifications.

Jonah has many years of working experience in different tech roles and industries. She has several years of experience in software development in C# .NET and cloud development with Microsoft Azure. Furthermore, she likes solving challenging, complex problems, programming back-end, and software engineering, and cloud development in .NET with cloud computing technologies like Microsoft Azure.

Jonah likes continuous learning, connecting with other developers, and sharing her technical knowledge at her workplace and through public speaking.

She is the founder of Azure User Group Sweden, a global Azure Users Group Community in Sweden that is an open and inclusive tech community for cloud development in Azure. Jonah organizes the technical sessions related to Microsoft Azure cloud services and is also active in sharing her knowledge through this community group. 

Jonah’s mission is to inspire tech to others. She wants to make a difference in the tech world by being a role model, advocating gender equality, diversity and inclusion in tech, being a mentor to youth and young women, inspiring tech and cloud technologies and helping them choose a tech career path in software engineering. 

If you want to directly reach out and connect to Jonah Andersson,
please use the following contact information:
Website: https://www.jonahandersson.tech
LinkedIn: https://www.linkedin.com/in/jonahandersson/
Twitter: https://www.twitter.com/cjkodare
GitHub: https://github.com/jonahandersson