Book cover of Beginning DevOps on AWS for iOS Development

Abdullahi Olaoye

Beginning DevOps on AWS for iOS Development

Xcode, Jenkins, and Fastlane Integration on the Cloud

Logo of the publisher

Abdullahi Olaoye

McKinney, TX, USA

ISBN 978-1-4842-8022-5e-ISBN 978-1-4842-8023-2

https://doi.org/10.1007/978-1-4842-8023-2

© Abdullahi Olaoye 2022

This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This Apress imprint is published by the registered company APress Media, LLC part of Springer Nature.

The registered company address is: 1 New York Plaza, New York, NY 10004, U.S.A.

I dedicate this book to my loving wife Aishat and lovely children Asiya and Ayyub, who have always supported and loved me unconditionally.

Any source code or other supplementary material referenced by the author in this book is available to readers on the Github repository: https://github.com/Apress/Beginning-DevOps-on-AWS-for-iOS-Development. For more detailed information, please visit http://www.apress.com/source-code.

Table of Contents

Chapter 1:​ Basics of iOS Application Development1

Development Requirements1

Mac2

Apple Developer Account2

Swift4

Introduction to Xcode4

Creating an Application6

Building an Application16

Xcode Command Line Tool20

Summary23

Chapter 2:​ From Xcode to App Store Connect25

Identifiers25

App Store Connect28

Screenshot and App Details31

Uploading Build from Xcode33

Testing and Publishing40

TestFlight40

App Submission43

Summary44

Chapter 3:​ DevOps on Amazon Web Services (AWS)45

Continuous Integration on AWS46

AWS CodeCommit46

AWS CodeBuild50

AWS CodeArtifact56

Continuous Delivery on AWS59

AWS CodeDeploy59

AWS CodePipeline62

Infrastructure As Code68

AWS CloudFormation68

AWS Cloud Development Kit (CDK)74

Monitoring and Logging80

Amazon CloudWatch80

AWS CloudTrail82

Summary83

Chapter 4:​ macOS Server on AWS85

Amazon EC2 Mac Server85

EC2 Limits and Pricing86

Deploying Amazon EC2 Mac Server88

Deploying with AWS Management Console88

Deploying with AWS CLI96

Deploying with AWS CloudFormation97

Deploying with Terraform102

Connecting to the Amazon EC2 Mac Server106

Connecting with SSH106

Connecting with VNC Client109

Monitoring EC2 Mac Server with Amazon CloudWatch114

Cleaning Up Amazon EC2 Mac Server116

Summary116

Chapter 5:​ Setting Up Development Tools on macOS Instance117

Increasing macOS Instance Volume Size117

Setting Up Xcode121

Xcode Command Line Tools123

Setting Up Jenkins124

Provisioning Jenkins Controller124

EC2 Mac Instance As Jenkins Build Agent130

Setting Up Fastlane138

Setting Up GitLab140

Creating GitLab Project140

Setting Up macOS GitLab Runner142

Summary146

Chapter 6:​ Source Control Management with AWS CodeCommit147

Git Fundamentals148

Git Installation148

Initializing a Git Repository149

Recording Changes to Git Repository149

Cloning and Working with Remote Git Repositories151

Git Branch153

Pull Requests154

Creating an AWS CodeCommit Repository154

Creating with AWS Command Line Interface (CLI)154

Creating with AWS CloudFormation156

Creating with Terraform158

Creating with Python Boto3 SDK162

Adding Source Code in AWS CodeCommit164

Connecting to a CodeCommit Repository165

Pushing Code to CodeCommit Repository175

Migrating Git Repository to AWS CodeCommit177

Branching in AWS CodeCommit180

Creating Branch on AWS Console180

Creating Branch with Git182

Creating Branch with AWS CLI183

Updating Local Repository185

Configuring Default Branch185

Pull Requests in AWS CodeCommit186

Creating a Pull Request187

Merging Pull Request191

Creating Approval Rule Template195

Pull Request with Approval Rule Template199

Amazon CodeGuru Reviewer Integration201

Summary205

Chapter 7:​ Integrating AWS CodeCommit with Jenkins207

Jenkins CodeCommit Plugin207

Setting Up Integration Components210

Setting Up Components via AWS Console211

Setting Up Components via Terraform222

Configuring Plugin226

Creating Jenkins Job with AWS CodeCommit Source229

Testing AWS CodeCommit Plugin234

Summary236

Chapter 8:​ Automated Build, Test, and Release with Fastlane237

Setting Up Code Signing with Fastlane match and Amazon S3238

Initializing Fastlane match239

Storing Certificates and Keys in Amazon S3242

Storing match Secrets and Parameters in AWS Secrets Manager244

Setting Up Jenkins Environment246

Setting Up and Verifying Xcode Installation246

Verifying Fastlane Installation248

Setting Up AWS CLI249

Installing JSON Query Language (jq)249

Automating Test and Build with Fastlane249

Adding Lane to Fastfile249

Configuring Jenkins Project251

Triggering Jenkins Project253

Automating Release to App Store Connect256

Setting Up App Store Connect API Key257

Storing Secrets in AWS Secret Manager260

TestFlight Release262

Automating Release to the App Store267

Summary271

Chapter 9:​ Testing with AWS Device Farm273

Introduction to AWS Device Farm273

Generating ipa Package for App Testing275

Setting Up a Device Farm Project and Scheduling Test Runs277

Scheduling Test Runs with AWS Console277

Scheduling Test Runs with AWS CLI289

AWS Device Farm Jenkins Plugin301

Installing Plugin301

Configuring Plugin302

Automating AWS Device Farm Tests with Jenkins307

Configuring Jenkins Project307

Triggering Jenkins Project311

Summary313

Chapter 10:​ Continuous Delivery Pipelines for iOS App Development315

Jenkins Pipeline316

Fastlane Test Stage321

AWS Device Farm Test Stage324

Fastlane Build Stage329

Fastlane Deliver Stage332

AWS CodePipeline336

Setting Up Jenkins Environment for AWS CodePipeline337

Setting Up CodePipeline on AWS Console344

Fastlane Testing with AWS CodePipeline and Jenkins348

AWS Device Farm Integration357

Summary367

Index369

About the Author

Abdullahi Olaoye

works at Amazon Web Services (AWS) as a Cloud Architect, where he is a technical leader and specializes in DevOps and Cloud Architecture. He earned a Master of Science in Computer Networking from Wichita State University, Kansas, USA, and a Bachelor of Engineering from All Nations University, Ghana. As a DevOps and cloud expert, he works at the forefront of cloud innovations by assisting enterprises of different scales in designing, building scalable IT solutions that solve business challenges, and in running their workloads on AWS Cloud by helping automate deployments and providing architectural guidance following best practices.

Abdullahi has authored and contributed to numerous technical AWS public content, such as AWS Blog posts, whitepapers, and AWS Prescriptive Guidance, covering topics from continuous deployment at scale to general architecture patterns.

He is very passionate about cloud computing, and he holds seven AWS certifications, which include the AWS Certified Solutions Architect Professional and AWS Certified DevOps Engineer Professional. He is also a Google Cloud Certified Associate Cloud Engineer. In his spare time, he enjoys listening to history podcasts and traveling. You can follow him on LinkedIn and GitHub at https://github.com/abdulola.

About the Technical Reviewer

Syed Nabeel

is an IT professional with 13+ years of experience and has been working at Amazon Web Services since April 2019 as Cloud Infrastructure Architect. Before joining AWS, Syed worked as Senior Software Engineer for GameStop, Verizon Wireless, and AllState Insurance. Syed loves learning new technologies, reading books, playing soccer/video games, and exploring new places with his family. He is also an entrepreneur and completed his undergraduate studies from Wright State University in MIS and Accounting and his Master of Science in Business Informatics from Northern Kentucky University.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

A. OlaoyeBeginning DevOps on AWS for iOS Developmenthttps://doi.org/10.1007/978-1-4842-8023-2_1

1. Basics of iOS Application Development

Abdullahi Olaoye1  

(1)

McKinney, TX, USA

This chapter is intended for beginners in iOS application development. If you are an iOS developer or have experience developing iOS applications, it can be skipped.

To develop iOS apps, Apple provides several tools and resources. iOS apps can be developed in native programming languages such as Swift or Objective-C and other cross-platform languages.

In this chapter, I will cover the requirements and basic building blocks of iOS application development while also walking you through the procedure by building a sample application with Swift and testing the application on an iOS device and iOS simulators.

Development Requirements

As an iOS developer, you need a macOS device and an Apple developer account to start development.

Mac

To start iOS development, a Mac is required. It is available in both notebook and desktop versions. The current notebook versions are MacBook Air and MacBook Pro, and the desktop versions are iMac, Mac Pro, and Mac mini. Some of the Mac models can be seen in Figure 1-1.

Figure 1-1

Comparing various Mac models

All the various Mac options can be compared at www.apple.com/mac/compare/.

Apple Developer Account

You can register as an Apple developer for free. All you need is an Apple ID. With the free developer account, you will have access to install Xcode on your Mac (which we will cover in the next section), iOS development documentations, sample code, Apple developer forums, and bug reporting. You will also be able to test your apps on devices. The free Apple developer account is sufficient for local testing and development. However, to distribute apps, you will need to enroll in the Apple Developer Program.

The Apple Developer Program is a paid membership program which is billed yearly. It is required if you are interested in creating apps for distribution on the App Store. More details on the Apple Developer Program can be found on the Apple website as seen in Figure 1-2.

Figure 1-2

Apple Developer Program

The Apple developer website can be accessed at https://developer.apple.com/programs/.

Swift

Swift is a programming language developed by Apple in 2014. The aim of developing Swift was to find a replacement for C-based languages such as C, C++, and Objective-C. With the expressive nature of its syntax, Swift has modern features developers want, and it makes iOS application development easier.

There are various resources available to learn Swift; however, Swift Playgrounds, an app developed by Apple, allows you to learn Swift in a fun and interactive way. Currently the app is available for iPad and Mac and can be downloaded from the App Store, as seen in Figure 1-3.

Figure 1-3

Swift Playgrounds

Introduction to Xcode

In addition to enabling development on MacOS systems, Xcode is an integrated development environment (IDE) developed by Apple and it has all you need to start iOS development across all Apple OS platforms. It includes features such as a code editor, debugger, etc. It also comes with a simulator which enables you to build and test your application on real iOS devices without needing a physical iOS device.

In the following sections, I will show you how to install Xcode and use it to build and deploy an iOS application.

Installing Xcode 12

I will be installing Xcode 12 on macOS Big Sur, which are currently the latest versions of Xcode and macOS, respectively. To install, go to the App Store on your Mac and search for Xcode, as seen in Figure 1-4.

Figure 1-4

Xcode search result on the App Store

Click on the install icon and wait for installation to complete.

If you get an error like “Xcode can’t be installed on “Macintosh HD” because macOS version X or later is required," update your macOS before trying to install.

After installation, launch Xcode and you should see a welcome page, as seen in Figure 1-5.

Figure 1-5

Launching Xcode

Next, I will show you how to use Xcode for app creation by creating a sample application.

Creating an Application

With Xcode installed, you are ready to start building iOS apps. I will start by using an app template which is provided by Xcode and then add a custom code to customize the application.

iOS Application Template

Xcode provides sample templates that make it easy to get started building apps. These templates can be accessed by creating a new Xcode project (Figure 1-5) and the different available templates are displayed. For the sample app I am building, I will choose iOS and App, as shown in Figure 1-6.

Figure 1-6

Choosing template for Xcode project

If this is your first time using Xcode, there will be a prompt to sign in with an Apple ID, as shown in Figure 1-7.

Figure 1-7

Apple sign in for new Xcode project

After signing in and linking an Apple ID to Xcode, there will be a prompt to configure the details for your new Xcode project. An example is shown in Figure 1-8.

Figure 1-8

Filling in details for new Xcode project

A folder location on the workstation to save the app must be selected before creation. Once complete, an app is created, as seen in Figure 1-9.

Figure 1-9

Sample app project template created

As seen in Figure 1-9, the application template is written in Swift language since Swift was selected as the language when creating the project in Figure 1-8.

Let us look at the out-of-the-box Swift code and see what it is doing.

import SwiftUI

struct ContentView: View {

    var body: some View {

        Text("Hello, world!")

            .padding()

    }

}

struct ContentView_Previews: PreviewProvider {

    static var previews: some View {

        ContentView()

    }

}

Listing 1-1

ContentView.swift

The code snippet in Listing 1-1 is responsible for the view that shows up on the iOS device; as seen, this iOS app template simply prints out “Hello, world!”.

import SwiftUI

@main

struct SampleAppApp: App {

    var body: some Scene {

        WindowGroup {

            ContentView()

        }

    }

}

Listing 1-2

SampleAppApp.swift

Listing 1-2 shows the entry point of the app, and as seen, it runs the application by calling ContentView() (defined in a different file ContentView.swift), which is shown in Listing 1-1.

Xcode Live Preview

Selecting SwiftUI interface when configuring the project enables live previews of the app as changes are being made to the content view in code.

Let us look at a preview of the default app. From the run destination menu, select the iOS simulator you want to use, as seen in Figure 1-10.

Figure 1-10

Selecting iOS simulator

The iPhone 11 simulator is selected, and the preview is shown on a simulated iPhone 11 as shown in Figure 1-11. As shown, live preview gives you the ability to see what the app view looks like before running a build. As changes are made to the app view in code, the preview is also updated.

Figure 1-11

Default app preview on iPhone 11 simulator

Developing with Swift

Until this point, you have been working with the default app provided as part of the template selected when creating an Xcode project, now let’s add a custom Swift code.

Replace the content of ContentView.swift with the code in Listing 1-3.

The code can also be downloaded at https://github.com/abdulola/devops-on-aws-ios-development-app/blob/master/SampleApp/ContentView.swift

import SwiftUI

struct ContentView: View {

    @State private var date = Date()

    let dateRange: ClosedRange<Date> = {

        let calendar = Calendar.current

        let startComponents = DateComponents(year: 2021, month: 1, day: 1)

        let endComponents = DateComponents(year: 2021, month: 12, day: 31, hour: 23, minute: 59, second: 59)

        return calendar.date(from:startComponents)!

            ...

            calendar.date(from:endComponents)!

    }()

    var body: some View {

        DatePicker(

            "Start Date",

            selection: $date,

            in: dateRange,

            displayedComponents: [.date, .hourAndMinute]

        )

        .padding(.all)

        .datePickerStyle(GraphicalDatePickerStyle())

    }

}

struct ContentView_Previews: PreviewProvider {

    static var previews: some View {

        ContentView()

    }

}

Listing 1-3

Sample Application Code

The sample app is a simple dummy calendar app that lets you select a date and a time within that date. Next, you will add an icon for this application.

Adding App Icon

An app icon is the visual representation of an app that is displayed on the home screen, the App Store, and various other places such as settings and search results. Due to the varied usage, different sizes of the app icon are required by Apple.

Example icons for this app can be downloaded from GitHub at https://github.com/abdulola/devops-on-aws-ios-development/tree/main/Chapter_1/AppIcons.

App icons can be added by accessing Assets.xcassets and dragging the icons into their respective slots, as shown in Figure 1-12.

Figure 1-12

Adding app icons

Building an Application

We’ve created a custom sample application code and added an icon for the application. We have all the basic building blocks of the application in place. Now we will build the application and run it on iOS devices.

Building on a Physical iPhone

Let us look at what the application looks like on a physical iOS device, such as an iPhone.

To build an application on a physical iPhone, it must be plugged into the Mac workstation used for development. Once the device is plugged into the workstation, it can be selected as a build destination. An example is shown in Figure 1-13.

Figure 1-13

Selecting physical iOS device for building

To start building on a connected physical iOS device, click the play sign button to the left of the device selection (Figure 1-13). An iOS keychain prompt might be shown, as seen in Figure 1-14. This requires the Mac workstation password (not your Apple ID password).

You will be required to add your Apple account to Xcode under Xcode ➤ Preferences to set up an Apple development team.

Make sure your iPhone is unlocked for Xcode to connect to it and install the Sample App. If this your first time plugging your iPhone to the Mac workstation, you will get a prompt to trust the workstation.

You might also be required to trust the app developer from your iPhone by going to Settings ➤ General ➤ Device Management.

Figure 1-14

Apple keychain login prompt

Once the build is complete, the new app with its icon can be found installed on the iPhone.

Figure 1-15 shows an example UI of the app when launched on a physical iPhone.

Figure 1-15

Sample app launch on a physical iPhone

Building on an iOS Simulator

Just like building and running the application on a physical device, let’s run the application on a simulator.

To build on a simulator, the target device simulator should be selected just like how the physical iOS device was selected, and to start building, click the play sign button to the left of the device selection, as shown in Figure 1-13.

Once the build is complete, the simulator will show the simulated iPhone with the app installed on it. Use your cursor to scroll through the pages of the simulated iPhone and launch the installed application. Figure 1-16 shows the icon of the application and what it looks like when launched on an iPhone 11 simulator.

Figure 1-16

Sample app launch on iPhone 11 simulator

Xcode Command Line Tool

So far, I have shown you how to build applications using the Xcode user interface; however, you can also interact with Xcode through the command line interface using its command line tool xcodebuild .

xcodebuild allows you to perform tasks such as building, testing, and archiving on your Xcode projects from the command line. With it, you can perform all the tasks we did earlier from the user interface on the command line, which opens the door for automation.

It comes preinstalled with Xcode so there is no additional action to install it on your macOS workstation. Run xcodebuild -usage command on your workstation to see some basic usage information, as seen in Listing 1-4.

$ xcodebuild -usage

Usage: xcodebuild [-project <projectname>] [[-target <targetname>]...

       xcodebuild [-project <projectname>] -scheme <schemeName> [-destination <destinationspecifier>]...

       xcodebuild -workspace <workspacename> -scheme <schemeName> [-destination <destinationspecifier>]...

       xcodebuild -version [-sdk [<sdkfullpath>|<sdkname>] [-json] [<infoitem>] ]

       xcodebuild -list [[-project <projectname>]|[-workspace <workspacename>]] [-json]

       xcodebuild -showsdks [-json]

Listing 1-4

xcodebuild -usage truncated output

To see more usage details including all available options, run xcodebuild -help and see the output, shown in Listing 1-5.

$ xcodebuild -help

Usage: xcodebuild [-project <projectname>] [[-target <targetname>]...

...

       xcodebuild -version [-sdk [<sdkfullpath>|<sdkname>] [-json] [<infoitem>] ]

       xcodebuild -list [[-project <projectname>]|[-workspace <workspacename>]] [-json]

       xcodebuild -showsdks [-json]

...

Options:

    -usage                        print brief usage

    -help                         print complete usage

    -verbose                      provide additional status output

    -license                      show the Xcode and SDK license agreements

    -checkFirstLaunchStatus       Check if any First Launch tasks need to be performed

    -runFirstLaunch               install packages and agree to the license

    -project NAME                 build the project NAME

    -target NAME

...

Listing 1-5

xcodebuild -help truncated output

You can use xcodebuild to navigate through the sample application created in the earlier sections, as seen in Listings 1-6 and 1-7.

First, you can list all the information about the sample application project.

You must be within the project and workspace directory to run the following commands.

$ xcodebuild -list -project SampleApp.xcodeproj

Command line invocation:

    /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -list -project SampleApp.xcodeproj

User defaults from command line:

    IDEPackageSupportUseBuiltinSCM = YES

Information about project "SampleApp":

    Targets:

        SampleApp

        SampleAppTests

        SampleAppUITests

    Build Configurations:

        Debug

        Release

    If no build configuration is specified and -scheme is not passed then "Release" is used.

    Schemes:

        SampleApp

Listing 1-6

Listing all the information about a project

Next, you can list the schemes in your workspace.

$ xcodebuild -list -workspace project.xcworkspace

Command line invocation:

    /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -list -workspace project.xcworkspace

User defaults from command line:

    IDEPackageSupportUseBuiltinSCM = YES

Information about workspace "SampleApp":

    Schemes:

        SampleApp

Listing 1-7

Listing all schemes in your workspace

Summary

This chapter showed you the basic building blocks of iOS application development; you got hands-on experience by building your own application and deploying it onto an actual iOS device and iOS simulator.

In the next chapter, we will take this one step further and explore how you can submit this built application to the App Store Connect for distribution and/or eventual submission to Apple’s App Store.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

A. OlaoyeBeginning DevOps on AWS for iOS Developmenthttps://doi.org/10.1007/978-1-4842-8023-2_2

2. From Xcode to App Store Connect

Abdullahi Olaoye1  

(1)

McKinney, TX, USA

This chapter is intended for beginners in iOS application development; if you are an iOS developer or have experience developing iOS applications, it can be skipped.

In the previous chapter, we saw how to use Xcode to build and run an application on a physical iPhone device and a simulator. We will take this one step further by exploring how to distribute the previously built application by uploading to App Store Connect.

An Apple Developer Program account covered in Chapter 1 (Figure 1-2) is required to perform the steps in this chapter.

Identifiers

Before an application build can be uploaded to App Store Connect, an identifier must be registered for the application. The identifier is used to uniquely identify an application so that no two applications share an identifier.

A new identifier can be created from the Apple Developer Program portal. From the portal, you can access the Certificates, Identifiers & Profiles page, as shown in Figure 2-1.

Figure 2-1

App Developer Program portal

The portal can be accessed at https://developer.apple.com/.

A new identifier can be created, as shown in Figure 2-2.

Figure 2-2

Creating a new identifier

There are different types of identifiers one can create, as shown in Figure 2-3. An App ID should be created.

Figure 2-3

Selecting identifier type

There are currently two types of App IDs. Select a type as shown in Figure 2-4.

Figure 2-4

Selecting App ID type

A description and ID name should be provided. An example is shown in Figure 2-5.

Take note of the ID name you used here.

Figure 2-5

Providing ID name and description

App Store Connect

App Store Connect is a platform for managing everything about an application on the App Store. Most importantly, applications are submitted to Apple’s App Store through App Store Connect. You get access to App Store Connect by having an Apple Developer Program account. The portal is shown in Figure 2-6.

App Store Connect can be accessed at https://appstoreconnect.apple.com.

Figure 2-6

App Store Connect

Before you can upload your application content to App Store Connect, the application must first be created. This will serve as the placeholder for the application on the App Store and all information about the application can be configured here.

To create an application, navigate to App Store Connect Apps page and create an app as shown in Figure 2-7.

Figure 2-7

Creating a new app

The Apps page can be accessed at https://appstoreconnect.apple.com/apps.

You might see an option to create a new App or new App bundle. Select “New App.”

To create the app, select the bundle ID that was created earlier in Figure 2-5 and provide the application details. Figure 2-8 shows the application details I provided.

Your application name must be unique and must not already exist on the App Store.

Figure 2-8

Providing New App details

Once the application is created, a landing page is shown, like in Figure 2-9, where you can configure even more details about your app, such as screenshots and other metadata information.

Figure 2-9

New App in App Store Connect

Screenshot and App Details

Now that the app is created, you can go ahead to add screenshots of the application that will be displayed on the App Store and other relevant information for your application.

Add as many details for the application as you can. Some of the information to be entered for the app are screenshots, app category, pricing and availability, app privacy, version releases, etc., as we will see in the following figures.

While there are various tools for creating screenshots, the iOS simulator can be used to take screenshots when an app is run on a simulator. Figure 2-10 shows where screenshots are added on App Store Connect.

Figure 2-10

Adding a screenshot for the new app in App Store Connect

Example screenshots created from the iOS simulator can be downloaded from https://github.com/abdulola/devops-on-aws-ios-development/tree/main/Chapter_1/Screenshots.

Figure 2-11 shows an example of how you can configure the category of your application. The app category can be accessed by selecting General ➤ App Information from the left menu.

Figure 2-11

Configuring category for your application

Figure 2-12 shows how you can configure the Pricing and Availability of your application.

Figure 2-12

Configuring pricing and availaibity for your application

Figure 2-13 shows how you can configure the Privacy Policy of your application.

Figure 2-13

Adding a privacy policy for your application

Figure 2-14 shows an example configuration for releasing new versions of your application.

Figure 2-14

Adding version release settings for your application

Uploading Build from Xcode

At this point, you have an app created in App Store Connect and have configured all the app metadata. Now you are ready to upload the build to App Store Connect from Xcode.

To do that, you will need to create a full app archive. We do this by selecting Any iOS Device from the list of simulators, as shown in Figure 2-15.

Figure 2-15

Selecting simulator for full app archive

Before creating the archive, verify that the Bundle Identifier configured on Xcode matches the Bundle ID configured on App Store Connect, as shown in Figure 2-5. This can be verified in the General settings of the app on Xcode, as shown in Figure 2-16.

Figure 2-16

Verifying bundle ID

Once the bundle ID is verified, you are ready to create a full archive. To create the archive, select Product ➤ Archive, as shown in Figure 2-17.

Figure 2-17

Creating full app archive

Once the archiving completes, a new window pops up, as shown in Figure 2-18, showing the archive that was created. Multiple archives will also be listed (if available), with the version and creation date of that archive.

Next, select Distribute App, as shown in Figure 2-18.

Figure 2-18

App archives to distribute

To distribute the app, you must select how to distribute it. Since the goal here is to upload to App Store, you select App Store Connect, as shown in Figure 2-19.

Figure 2-19

Uploading app to App Store Connect

To upload an iOS app to App Store Connect, an Apple Distribution certificate is required. If Xcode does not find a certificate, for example, if you’re uploading an app for the first time, it will attempt to create a new one for you, and you will get a prompt like in Figure 2-20.

Figure 2-20

Creating an Apple Distribution certificate

After Xcode generates the certificate, it stores it in Keychain but also gives you an opportunity to export the certificates p12 file for safekeeping. You will get a prompt like in Figure 2-21.

Exporting the certificate is not required but recommended. If you choose not to export the certificate, you can skip by clicking Next.

Figure 2-21

Exporting Apple Distribution certificate

Before upload starts, you will be prompted with App Store distribution options such as including bitcode for iOS content and uploading app symbols; select these options.

Additionally, you will be prompted to select the signing option. Select the option to have Xcode automatically manage signing.

If you already have a certificate and a provisioning profile to be used, you can manually manage signing.

Once the app upload starts, you should see a progress bar, like in Figure 2-22.

Figure 2-22

App upload in progress

Once the upload completes and is free of errors, you will see an upload success message, as shown in Figure 2-23.

Figure 2-23

App upload complete

Now that the upload is successful, allow some time for the app to be visible on App Store Connect, you will typically get an email notification when your build is ready on App Store Connect.

Next, we will explore how to do beta testing with the new app uploaded and, finally, how an app is submitted for review.

Testing and Publishing

When an uploaded build becomes available on App Store Connect, you can do an optional beta testing of the application before final submission to the App Store. Let’s explore how to accomplish these on Apple’s App Store Connect.

TestFlight

Beta testing is not a required step in iOS application development process. However, Apple provides a tool called TestFlight which can be used for beta testing. With TestFlight, you can invite actual users to download and install your app on their devices to receive feedback before it is publicly available.

To add testers for beta testing, you go the TestFlight tab and select the build. You have the option of adding individual testers or a group and provide information to testers, as shown in Figure 2-24.

Figure 2-24

Adding build test information

If you select the option to add individual testers, you will get a popup window, as shown in Figure 2-25, where you should provide the name and contact of the testers.

Figure 2-25

Adding testers to build

After adding testers, you add test information and submit the test for review, as shown in Figure 2-26.

Figure 2-26

Submitting beta test for review

App Submission

The final step in this process is app submission for review. Scroll to the Build section on App Store Connect, and to submit your app for review, first select a build that has been uploaded to App Store Connect, as shown in Figure 2-27.

Figure 2-27

Selecting an uploaded build

Once a build is selected and added, the Submit for Review button is enabled, as shown in Figure 2-28.

Figure 2-28

Submit app for review

Submitting the app here initiates a review by Apple before the app becomes available.

Summary

This chapter builds on the previous chapter to wrap up the basics of iOS development and has showed you how to interact with App Store Connect to upload application builds from your workstation, perform beta testing, and submit for review.

In the next chapter, we will go into DevOps by reviewing the fundamentals of DevOps, AWS Cloud and you will learn about the different AWS DevOps offerings.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

A. OlaoyeBeginning DevOps on AWS for iOS Developmenthttps://doi.org/10.1007/978-1-4842-8023-2_3

3. DevOps on Amazon Web Services (AWS)

Abdullahi Olaoye1  

(1)

McKinney, TX, USA

DevOps has been defined differently across the industry, but all these definitions mostly point to the same principles. AWS defines DevOps as follows:

DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes.

The primary goal of the DevOps concept is to improve software development lifecycle. To accomplish this, it introduces different methodologies, tooling, processes, etc.

AWS has a rich offering of tooling for various aspects of DevOps, it also integrates with third-party tools to give users a wide range of options for practicing DevOps on AWS. In this chapter, I will cover some basic AWS Cloud concepts, then we will explore the main tools and technology available on AWS for practicing DevOps. I will give an introduction into these services and cover hands-on demo on how to get started with these services on the AWS Management Console. We will also cover some common integration with third-party tools.

All example code used in this chapter can be found at https://github.com/abdulola/devops-on-aws-ios-development/tree/main/Chapter_3.

Continuous Integration on AWS

The concept of continuous integration is for developers to merge code regularly in a source control management tool and run frequent automated tests. This allows you to detect software issues quickly, thereby improving overall quality of software delivery.

Let us explore the different continuous integration tools provided by AWS in the following sections.

AWS CodeCommit

In a nutshell, AWS CodeCommit is a managed private git repository service. As a managed AWS service, you do not manage the underlying hardware or configuration to support hosting the private git repository hence it is secure and highly scalable. As a git repository, it supports standard git functionality, so it works with existing git-based tools such as git cli, GUI clients, etc.

Benefits of Using AWS CodeCommit

There are a lot of benefits when using AWS CodeCommit as your private git repository; all these are summarized into the five main points highlighted as follows:

1.

It is highly available and durable.

2.

Data is protected and secure, with encryption of data at rest and in transit.

3.

Fine-grained access control can be defined.

4.

Ease of collaboration between team members.

5.

Integrates with other AWS automation tools to extend its functionalities.

Creating a CodeCommit Repository

There are various ways to create a CodeCommit repository such as through the AWS Management Console, AWS CLI, AWS SDKs, etc.; however, I will show the simplest way to get started using the AWS management console interface.

First you login to the AWS Management Console and search for CodeCommit in the search bar, as seen in Figure 3-1.

Figure 3-1

AWS CodeCommit search on the AWS Management Console

Another option for launching the CodeCommit page is to select CodeCommit from the Developer Tools section of the AWS Management Console, as shown in Figure 3-2.

Figure 3-2

Selecting CodeCommit from the Developer Tools Section

After launching the CodeCommit web page, create a repository referencing Figure 3-3.

Figure 3-3

Starting CodeCommit repository creation

Provide the repository name, optional description, and select Create, as seen in Figure 3-4.

Figure 3-4

Creating CodeCommit repository

After repository creation, the next step is to connect to your repository and start using it, as seen in Figure 3-5. To connect, you use existing git-based tools.

We will cover CodeCommit connectivity in more detail in Chapter 6.

Figure 3-5

Connecting to a CodeCommit Repository

AWS CodeBuild

AWS CodeBuild is a managed build server on the cloud. It provides a compute environment to compile code and run tests and product artifacts. CodeBuild takes source code from a source provider and performs actions as defined in a YAML specification file usually called Buildspec. For source providers, CodeBuild currently supports Amazon S3, AWS CodeCommit, GitHub, Bitbucket, and GitHub Enterprise.

Creating a CodeBuild Project

From the AWS Management Console, search for CodeBuild in the search bar, as seen in Figure 3-6.

CodeBuild project can also be created from the AWS CLI, AWS SDKs, and other infrastructure as code tools.

Figure 3-6

AWS CodeBuild search on the AWS Management Console

You can also select CodeBuild from the Developer Tools section of the AWS Management Console, as shown in Figure 3-2.

On the CodeBuild page, create a build project, as seen in Figure 3-7.

Figure 3-7

Initiating CodeBuild project creation

There are different sources that are supported by a CodeBuild project. The source code that will be compiled or built by CodeBuild is to be stored in this source repository. For builds that do not require source code, a source is not required to be configured. An example of such builds is when the build is configured to connect to an external location to fetch the required resources for the build.

As seen in Figure 3-8, no source is configured for the project I am about to create.

Figure 3-8

Configuring source for CodeBuild project

As mentioned earlier, a CodeBuild project provides a compute environment to perform build actions using a container image. You have the option of using a managed AWS CodeBuild image or providing your custom docker image. As seen in Figure 3-9, I will be using a managed CodeBuild image.

Figure 3-9

Compute environment for CodeBuild project

A CodeBuild project requires build specifications. When using a source provider for the project, the YAML build specification file can be stored in the code repository and the name and location configured in the project, CodeBuild will by default look for the filename buildspec.yaml. The build specification commands can also be provided directly to the project, this is especially required if the project is not configured with any source, which is the case with the CodeBuild project we are creating.

The listing below shows an example build specification. Each phase defined is executed in sequence and the commands in each phase are also executed in sequence.

More information on writing a build specification file can be found at https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html.

version: 0.2

phases:

  install:

    commands:

      - echo Entered the install phase...

      - apt-get update -y

      - apt-get install -y maven

  pre_build:

    commands:

      - echo Entered the pre_build phase...

  build:

    commands:

      - echo Entered the build phase...

      - echo Build started on `date`

  post_build :

    commands:

      - echo Entered the post_build phase...

      - echo Build completed on `date`

Listing 3-1

CodeBuild build specification

Copy this example and provide as a build specification as seen in Figure 3-10.

You might need to switch to editor mode to see the page to enter the build specification.

Figure 3-10

CodeBuild build specification

Leaving the remaining settings as default, the CodeBuild project is created, as seen in Figure 3-11. The CloudWatch Logs option checked in Figure 3-11 ensures that build logs are streamed to Amazon CloudWatch for ease of access and open the door for further logs processing.

Figure 3-11

Creating a CodeBuild project

AWS CodeArtifact

AWS CodeArtifact is an artifact repository that is used to store, publish, and share software packages used in a software development process. It is a fully managed service; hence, there is no need to manage infrastructure and licenses.

With AWS CodeArtifact, you can either publish private packages to be used by developers or configure it to fetch from public artifact repositories. When configured to fetch from public artifact repositories, the added advantage of using AWS CodeArtifact is the ability to define access control and validate software packages. It works with common package managers such as Maven, Gradle, npm, yarn, twine, pip, and NuGet.

Creating a CodeArtifact Repository

Let us look at how you can create an AWS CodeArtifact repository. I will select CodeArtifact from the Developer Tools section of the AWS Management Console, as shown in Figure 3-2, then start repository creation, as shown in Figure 3-12.

Figure 3-12

Starting CodeArtifact repository creation

AWS CodeArtifact can also be created from the AWS CLI, AWS SDKs, and other infrastructure as code tools.

You will get the page seen in Figure 3-13 where you fill in the repository name. If this repository is to be configured to connect to an upstream public repository, the upstream repository can also be configured here, as seen in Figure 3-13.

Figure 3-13

Providing repository details

There is a concept of domains in CodeArtifact. This is a logical grouping of repositories and every CodeArtifact repository must be part of a domain. A domain can also be in another AWS account which makes it easy to manage lots of CodeArtifact repositories in a multi-account AWS environment. A domain can be created or selected, as seen in Figure 3-14.

Figure 3-14

Creating a repository domain

After creating a domain, the next step is to review and create the repository, as seen in Figure 3-15.

Figure 3-15

Reviewing and creating a CodeArtifact repository

For initial connection, connection instructions are shown for different common package managers, for example, npm after creation.

Continuous Delivery on AWS

As discussed in the previous section, continuous integration covers how developers merge code regularly in a source control management tool and run frequent automated tests. Continuous delivery takes this a step further by deploying the code changes. Code changes can be deployed to a dev and/or test environment, which ensures you always have a production-ready artifact that has been built and tested.

AWS CodeDeploy

AWS CodeDeploy is an automated software deployment service. With CodeDeploy, you can automate deployment to applications running on an Amazon Elastic Compute Cloud (Amazon EC2) instance, Amazon Elastic Container Service (Amazon ECS), on-premises servers, and AWS Lambda functions.

To use CodeDeploy, you define how the application should be deployed in a YAML specification file popularly called appspec.yaml ; CodeDeploy uses this specification file to fulfill application deployment. For Amazon EC2 and on-premises deployment, a CodeDeploy agent must be installed on the fleet of application servers for CodeDeploy to manage deployment, but for the other computing platforms such as Amazon ECS and AWS Lambda, CodeDeploy agent is not used because these are managed AWS services.

The application specification file takes different forms depending on the target compute platform, for example, Amazon EC2, AWS Lambda, etc. Let us look at examples of specification files for different compute platforms.

The hooks section in Listings 3-3 and 3-4 is only required if you are running validation scripts with AWS Lambda functions as part of the deployment. In these examples, I am referencing AWS lambda functions that should already exist within the same AWS account and AWS region with CodeDeploy.

Listing 3-2 shows an example specification file for deployment to an Amazon EC2 or on-premises server.

version: 0.0

os: linux

files:

  - source: Config/config.txt

    destination: /webapps/Config

  - source: source

    destination: /webapps/myApp

hooks:

  BeforeInstall:

    - location: Scripts/UnzipResourceBundle.sh

    - location: Scripts/UnzipDataBundle.sh

  AfterInstall:

    - location: Scripts/RunResourceTests.sh

      timeout: 180

  ApplicationStart:

    - location: Scripts/RunFunctionalTests.sh

      timeout: 3600

  ValidateService:

    - location: Scripts/MonitorService.sh

      timeout: 3600

      runas: codedeployuser

Listing 3-2

Example appspec for Amazon EC2 or on-premises deployment

Listing 3-3 shows an example specification file for a deployment to an AWS Lambda function

version: 0.0

Resources:

  - myLambdaFunction:

      Type: AWS::Lambda::Function

      Properties:

        Name: "myLambdaFunction"

        Alias: "myLambdaFunctionAlias"

        CurrentVersion: "1"

        TargetVersion: "2"

Hooks:

  - BeforeAllowTraffic: "ValidateBeforeTrafficShift"

  - AfterAllowTraffic: "ValidateAfterTrafficShift"

Listing 3-3

Example appspec for AWS Lambda deployment

Listing 3-4 shows an example specification file for a deployment to an AWS Lambda function.

version: 0.0

Resources:

  - TargetService:

      Type: AWS::ECS::Service

      Properties:

        TaskDefinition: "arn:aws:ecs:us-east-1:111222333444:task-definition/my-task-definition-family-name:1"

        LoadBalancerInfo:

          ContainerName: "SampleApplicationName"

          ContainerPort: 80

Hooks:

  - BeforeInstall: "ValidateBeforeInstall"

  - AfterInstall: "ValidateAfterInstall"

  - AfterAllowTestTraffic: "ValidateAfterTestTraffic"

  - BeforeAllowTraffic: "ValidateBeforeProdTraffic"

  - AfterAllowTraffic: "ValidateAfterProdTraffic"

Listing 3-4

Example appspec for Amazon ECS deployment

AWS CodePipeline

With AWS CodePipeline , you can orchestrate and automate all the steps and actions that make up the release process of your software in the order of stages and actions. CodePipeline integrates with a variety of AWS services and third-party tools which enable you to create pipelines that automate the entire software delivery process from source all the way to deployment.

Creating a CICD Pipeline with AWS CodePipeline

To start a Pipeline creation, I will select CodePipeline from the Developer Tools section of the AWS Management Console, as shown in Figure 3-2, then start pipeline creation, as shown in Figure 3-16.

Figure 3-16

Starting pipeline creation

AWS CodePipeline can also be created from the AWS CLI, AWS SDKs, and other infrastructure as code tools.

Since the pipeline will be performing actions on your behalf, it requires a service IAM role that will define the permissions the pipeline has within the AWS account. When creating a pipeline on the AWS Management Console, the required IAM role can be created for you, or you can select an existing IAM role, as seen in Figure 3-17.

Figure 3-17

Creating a CodePipeline service role

Now you start to define the structure of the pipeline. The first stage of the pipeline is the source stage which will define where the application source code is stored. For a source stage, AWS CodePipeline supports Amazon ECR, AWS CodeCommit, Amazon S3, and third-party tools such as Bitbucket and GitHub. In Figure 3-18, AWS CodeCommit is configured as the source of the pipeline.

Figure 3-18

Configuring AWS CodeCommit as pipeline source stage

The branch name provided here should match a valid branch in your CodeCommit repository; otherwise the pipeline will fail.

Next, we configure a build stage. For the build stage, CodePipeline supports AWS CodeBuild, CloudBees, Jenkins, and TeamCity. Figure 3-19 shows using AWS CodeBuild for the build stage.

Figure 3-19

Configuring AWS CodeBuild as pipeline build stage

After the build stage comes the deploy stage. In the deploy stage, built artifacts are retrieved from the build stage and then deployed. You can have multiple deploy stages which represents different environments.

CodePipeline integrates with a variety of AWS services and tools to support your application deployment.

Figure 3-20 shows the supported deployment providers when creating a deploy stage on the AWS Management Console.

Figure 3-20

CodePipeline peployment providers

While a CodePipeline requires at least two stages during initial creation, the deploy stage is not required and can be skipped during pipeline initial creation. See Figure 3-21 on how to skip the deploy stage.

Figure 3-21

Skipping adding the deploy Stage

After all stages have been configured and integrated with the required tools, the pipeline can be created. As seen in Figure 3-22, you can review and then create the pipeline.

Figure 3-22

Reviewing and creating a pipeline

After pipeline creation, it runs for the first time and executes all the stages configured. As seen in Figure 3-23, two stages are configured (Source and Build); the pipeline takes the source artifact from AWS CodeCommit and builds it using AWS CodeBuild.

Figure 3-23

A successful pipeline run

Infrastructure As Code

One of the key principles of DevOps is deployment automation, but this automation concept does not only apply to applications, but also infrastructure automation. Defining infrastructure as code like applications allows for automated infrastructure deployments which greatly reduces deployment time, limits human errors, and allows reusability. With infrastructure defined as code, source control can also be utilized to manage the lifecycle of infrastructures.

We will explore the infrastructure as code (IaC) offering of AWS which will help you understand how to easily automate the provisioning of resources on AWS.

AWS CloudFormation

AWS CloudFormation is a service that allows you to define AWS resources in a template which can be written in jSON or YAML language. In CloudFormation, you define your target architecture and the integration of all the different components in the template and then create a stack to provision the resources.

AWS CloudFormation can be assessed on the AWS Console, as seen in Figure 3-24.

Figure 3-24

Accessing CloudFormation service from the AWS Console

To create a CloudFormation stack, you must have a template that defines the resources you will provision. CloudFormation has different parts and a defined anatomy that must be adhered to to ensure that CloudFormation knows what to do about the lifecycle of a resource or resources when a stack is created with a template.

The template given in Listing 3-5 shows the definition of an S3 bucket with versioning enabled. Furthermore, the name of the S3 bucket is generated dynamically at runtime and it consists of the AWS Account number and AWS region where the stack will be created, then the bucket name is passed as an output of the CloudFormation stack.

AWSTemplateFormatVersion: '2010-09-09'

Description: 'Sample S3 Bucket with Versioning Enabled'

Resources:

  S3Bucket:

    Type: AWS::S3::Bucket

    Properties:

      BucketName: !Sub ${AWS::AccountId}-${AWS::Region}-bucket

      VersioningConfiguration:

        Status: Enabled

    DeletionPolicy: Retain

Outputs:

  BucketName:

    Value: !Ref S3Bucket

    Description: S3 Bucket Name

Listing 3-5

CloudFormation template

To create a CloudFormation stack from a defined template, after accessing CloudFormation console from the search bar or the AWS Management Console home page, select the option to create stack, as seen in Figure 3-25.

AWS CloudFormation stack can also be created from the AWS CLI, AWS SDKs, and other infrastructure as code tools.

Figure 3-25

Creating a CloudFormation stack

Figure 3-25 is shown when there are no existing CloudFormation stacks in the account and region. If there are existing CloudFormation stacks, they will be listed but the “Create stack” button will still be enabled.

When the Create stack action provides you with stack creation options to create new or existing resources, choose to create new resources

This starts the creation workflow, and the first step is to provide the template you want to launch. Figure 3-26 shows how to upload a template that is already authored.

The template shown in Listing 3-5 is saved in a file named s3.yaml and used for stack creation.

Figure 3-26

Providing CloudFormation template

After providing this template and selecting Next, CloudFormation processes the template and parses for error and invalid entries. Once the template is validated, you can configure stack name and parameters (if any), as seen in Figure 3-27.

Figure 3-27

Configuring stack details

Ideally, the next and final step is to create the stack. However, there is a concept in CloudFormation called Change Set, which allows you to visualize and inspect change that will be made to resources in your CloudFormation stack, that is, resources that will be added, deleted, or just modified. Figure 3-28 shows the selection of either proceeding to create the stack or creating a change set.

Figure 3-28

Creating a change set

If you just opt to create the stack, CloudFormation starts creating your resources. If you opt to create a change set, additional configuration such as change set name will be required before the change set is created. Figure 3-29 shows using default values for change set configuration.

Figure 3-29

Configuring a change set

Change set will allow you to see details of configuration changes that will be made. Figure 3-30 shows that only one resource, an S3 bucket, will be added. This is because it is an initial stack creation and the CloudFormation template in Listing 3-5 only has a single S3 resource defined.

Figure 3-30

Change set for stack creation

You can delete a change set if you’re not satisfied with the changes, and no action will take place. When a change set is executed, resource creation or update starts. Figure 3-31 shows a CloudFormation stack that has been created from the change set executed in Figure 3-30.

Figure 3-31

Stack creation complete

AWS Cloud Development Kit (CDK)

CDK is an open-source infrastructure as code framework that allows you to define AWS resources in familiar programming languages such as TypeScript, JavaScript, Python, Java, and C#. This makes it easier for developers to integrate infrastructure provisioning to existing software development lifecycle.

When infrastructure is defined in CDK and deployed, CDK uses CloudFormation to deploy the resources, so it takes care of converting the CDK script to a corresponding CloudFormation template to deploy the resources.

AWS CDK Installation

AWS CDK toolkit can be installed with the node package manager (npm). Listing 3-6 shows an example installation of the AWS CDK toolkit.

See https://docs.npmjs.com/downloading-and-installing-node-js-and-npm for how to install npm if not installed on your workstation.

$ sudo npm install -g aws-cdk

/usr/bin/cdk -> /usr/lib/node_modules/aws-cdk/bin/cdk

> aws-sdk@2.932.0 postinstall /usr/lib/node_modules/aws-cdk/node_modules/aws-sdk

> node scripts/check-node-version.js

+ aws-cdk@1.110.0

added 181 packages from 216 contributors in 8.404s

Listing 3-6

AWS CDK toolkit installation

After installing CDK, you test the installation by checking the CDK version using the command cdk --version.

The version may differ based on the time of installation.

Deploying Infrastructure with CDK

With CDK correctly installed, you can start creating your infrastructure in any of the supported CDK languages. In Listing 3-7, I show how you can initialize a CDK application in Python and a snippet of the output.

$ mkdir hello-cdk

$ cd hello-cdk

$ cdk init app --language python

Applying project template app for python

# Welcome to your CDK Python project!

This is a blank project for Python development with CDK.

The `cdk.json` file tells the CDK Toolkit how to execute your app.

This project is set up like a standard Python project. The initialization process also creates a virtualenv within this project, stored under the `.venv` directory. To create the virtualenv it assumes that there is a `python3` (or `python` for Windows) executable in your path with access to the `venv` package. If for any reason the automatic creation of the virtualenv fails, you can create the virtualenv manually.

To manually create a virtualenv on MacOS and Linux:

```

$ python3 -m venv .venv

```

Listing 3-7

Initialize CDK Application

When a Python CDK project is initialized, it creates a Python virtual environment as seen in Listing 3-7. It also creates a requirement.txt file where you define your application dependencies. I will activate the Python virtual environment and install dependencies as seen in Listing 3-8.

$ source .venv/bin/activate

(.venv) $ python -m pip install -r requirements.txt

Listing 3-8

Installing CDK project dependencies

In CDK, infrastructure is defined in stacks and a CDK app must consist of at least one stack for resources to get deployed. When CDK converts its script to CloudFormation, it creates a CloudFormation stack per CDK stack.

A sample stack HelloCdkStack was created when I initialized the CDK app, and you can view a list of CDK stacks by running the command cdk ls, as seen in Listing 3-9.

(.venv) $ cdk ls

HelloCdkStack

Listing 3-9

CDK Stack List

For CDK to deploy resources, you must define these resources in the chosen programming language. So far, the CDK app initialized does not do anything because no resources have been defined. I will show you how to define resources by creating an S3 bucket in the following steps.

First, Listing 3-10 shows how to install the Amazon S3 CDK package from the AWS Construct library. The CDK package for every AWS resource that will be created must first be installed. Installing CDK does not install resource-level packages; these must be installed separately.

(.venv) $ pip install aws-cdk.aws-s3

Listing 3-10

Installing Amazon S3 Package

After all the required packages are installed, you can start defining resources and their properties in code. Listing 3-11 shows the definition of the S3 resource in the CDK stack.

Since the CDK app is named hello-cdk from Listing 3-7, the Python file location where the stack is defined will be hello_cdk/hello_cdk_stack.py.

#hello_cdk/hello_cdk_stack.py

from aws_cdk import core as cdk

from aws_cdk import aws_s3 as s3

class HelloCdkStack(cdk.Stack):

    def __init__(self, scope: cdk.Construct, construct_id: str, **kwargs) -> None:

        super().__init__(scope, construct_id, **kwargs)

        bucket = s3.Bucket(self, "MyFirstBucket",

            versioned=True,)

Listing 3-11

Amazon S3 CDK stack

After defining resources in CDK, the app is ready to be deployed. Before deployment, you can synthesize and see the resulting CloudFormation template for the CDK resources defined, as seen in Listing 3-12.

(.venv) $ cdk synth

Resources:

  MyFirstBucketB8884501:

    Type: AWS::S3::Bucket

    Properties:

      VersioningConfiguration:

        Status: Enabled

    UpdateReplacePolicy: Retain

    DeletionPolicy: Retain

    Metadata:

      aws:cdk:path: HelloCdkStack/MyFirstBucket/Resource

Listing 3-12

CDK synth output snippet

For deploying a CDK app, the command is cdk deploy. See Listing 3-13 for an example deployment.

(.venv) $ cdk deploy

HelloCdkStack: deploying...

HelloCdkStack: creating CloudFormation changeset...

 ✅  HelloCdkStack

Stack ARN:

arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/HelloCdkStack/36fb9a60-d6dc-11eb-8377-0ab7c9206afb

Listing 3-13

CDK app deployment

When a CDK app is deployed, it creates a CloudFormation stack, this stack can also be accessed on the AWS Management console as seen in Figure 3-32.

Figure 3-32

CloudFormation stack of the deployed CDK app

The CDK CLI can also be used to delete deployed resources. When a cdk application is deleted, the underlying CloudFormation stack is also deleted. Listing 3-14 shows the deletion of the single CDK stack that was created in Listing 3-13.

(.venv) $ cdk destroy

Are you sure you want to delete: HelloCdkStack (y/n)? y

HelloCdkStack: destroying...

 ✅  HelloCdkStack: destroyed

Listing 3-14

Deleting CDK app

Monitoring and Logging

Monitoring is an important component of DevOps. Having visibility into the performance of the deployed application and logging application activities allows you to proactively identify issues and patterns to further improve software quality.

Let’s explore the AWS offerings that can help you satisfy logging and monitoring requirements in your application starting with AWS CloudWatch.

Amazon CloudWatch

Amazon CloudWatch is the AWS offering for monitoring and logging of application performance and activity. Since there are different components to monitoring and logging, CloudWatch is divided into different subservices based on functionality. Figure 3-33 shows the CloudWatch Console. The different offerings of CloudWatch are shown in the left pane.

Figure 3-33

AWS CloudWatch service console

Let us explore some of the common use cases of AWS CloudWatch.

CloudWatch Metrics

Most AWS services natively publish metric data to CloudWatch metrics. When you create an AWS resource that has native integration with CloudWatch, without additional action, you can view metrics for that resource to monitor different aspect of the resource. AWS has out-of-the-box metrics supported for each resource type; however, you can also publish custom metrics to monitor components that are not natively supported by CloudWatch.

See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html for more information on publishing custom metrics to CloudWatch.

Published metrics can be viewed in a graph on the AWS console, as seen in Figure 3-34.

Figure 3-34

Available CloudWatch metrics

CloudWatch Logs

This is the log aggregation feature of AWS CloudWatch. Logs stored in AWS CloudWatch can be from your application running on-premises or on the cloud. Some AWS services have native integration with CloudWatch Logs and store their logs there with little to no additional configuration. Examples of such services are serverless products such as AWS Lambda, which is a service that lets you execute code on the cloud without provisioning resources, and AWS CodeBuild, which provides a compute environment to compile code, run tests, and product artifacts.

CloudWatch Logs also has a feature CloudWatch Logs Insights, which allows you to easily query and analyze your log data to respond to operational issues quickly.

CloudWatch Alarms

From Metric data present in CloudWatch, you can create alarms based on this data to perform actions when certain thresholds are met. For example, you can create a CloudWatch alarm to perform scaling actions when CPU utilization metric crosses a threshold. Metrics can be created from CloudWatch Logs data events, so this gives the ability to also configure alarms based on events in application logs stored on CloudWatch Logs.

AWS CloudTrail

All activities you perform on AWS either through the AWS Console, CLI, or SDK tools are API calls that are monitored and logged by AWS CloudTrail. CloudTrail logs are stored in an Amazon S3 bucket. The information stored by CloudTrail provides audit trail of activities in your AWS account, which makes it easier to audit actions of users or AWS services that perform actions on behalf of users.

Summary

In this chapter, I talked about the different AWS DevOps offerings for continuous integration, continuous delivery, infrastructure as code, and monitoring. We explored basic details, their use cases, and walked through example provisioning of these resources on the AWS Management Console.

In the next chapter, we will cover details of provisioning a macOS server on AWS, which will provide the foundation for developing iOS natively on AWS.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

A. OlaoyeBeginning DevOps on AWS for iOS Developmenthttps://doi.org/10.1007/978-1-4842-8023-2_4

4. macOS Server on AWS

Abdullahi Olaoye1  

(1)

McKinney, TX, USA

Traditionally, iOS application developers must purchase hardware macOS workstations or servers to develop iOS applications because these apps can only be built, tested, and deployed on an iOS operating system. This limits developer’s ability to integrate cloud native solutions into their development process, thereby not harnessing the power of the cloud.

The Amazon EC2 Mac instance removed these limitations, and in this chapter, we will dive into the different options of deploying it to start using it for your iOS app development.

All example code used in this chapter can be found at https://github.com/abdulola/devops-on-aws-ios-development/tree/main/Chapter_4.

Amazon EC2 Mac Server

Amazon EC2 Mac unlocks the ability to run on-demand macOS workloads on the cloud hence allowing iOS developers benefit from the flexibility, scalability, agility, and cost benefits the AWS cloud provides.

It is powered by Mac mini hardware and currently by 3.2 GHz Intel eighth generation (Coffee Lake) Core i7 processors. You can use EC2 Mac instances for developing, building, testing, and signing applications for Apple devices, such as iPhone, iPad, iPod, Mac, Apple Watch, and Apple TV.

To connect to an EC2 Mac instance, you can use SSH for command terminal access or a client that supports the Virtual Network Computing (VNC) protocol to get the full user interface (UI) experience.

EC2 Limits and Pricing

Because each EC2 Mac instance is powered by an actual Mac mini hardware behind the scenes, by default AWS places limitations on accounts from provisioning a Mac instance. This is to prevent abuse and preserve capacity for real users.

Before you attempt to deploy, review your EC2 limits to ensure you have the quota to deploy for your account in the region you intend to use. To check your quota, go to the EC2 Service from the AWS management console as seen in Figure 4-1.

Figure 4-1

Accessing EC2 from the AWS Management Console

From the EC2 Console, go to Limits as seen in Figure 4-2.

Figure 4-2

Accessing the Limits page from EC2 Console

There you can search for the Mac Dedicated Host limit, as shown in Figure 4-3. The limit indicates how many Mac EC2 instances you can provision. If applicable, you can raise a limit increase request from the portal and provide your use case.

Figure 4-3

EC2 Mac limit for account and region

Mac instances are only available as bare metal instances on Dedicated Hosts because there is a one-to-one mapping between an Amazon EC2 Mac instance and a Mac mini server. You can launch one Mac instance per Dedicated Host and you can share the Dedicated Host with the AWS accounts or organizational units within your AWS organization, or the entire AWS organization.

EC2 Mac instance is not eligible for free tier and is billed per allocated dedicated host, not per instance and the dedicated host has a minimum allocation period of 24 hours before you can release.

Since you cannot release the host until 24 hours have passed since you first allocated the host, you will be charged for at least 24 hours of usage.

See https://aws.amazon.com/ec2/instance-types/mac/ for more information on pricing.

Deploying Amazon EC2 Mac Server

Like most AWS services, there are various options available to you to provision an EC2 Mac instance. In this section, I will show you the main options available by walking through the creation and providing example code.

Deploying with AWS Management Console

The AWS Management Console is the easiest way to getting started provisioning AWS services. To provision an EC2 Mac Server, you will first provision a dedicated host. See Figure 4-4 on how to access Dedicated Hosts from the EC2 Console.

Figure 4-4

Accessing Dedicated Hosts from the EC2 Console

On the Dedicated Hosts console, you select the option to start the allocation as seen in Figure 4-5.

Figure 4-5

Starting allocation of a dedicated host

Fill in the allocation parameters as shown in Figure 4-6 and allocate a dedicated host. The main parameter fields are described as follows:

Name tag – Provide a name for the dedicated host.

Instance family – Specifies the instance family to be supported by the dedicated host. Here, select the Instance family “mac1” from the drop-down list.

Support multiple instance types – If you enable support for multiple instance types, the host can support different instance types in the selected instance family.

By default, this option will be grayed out for the Mac instance family because there is currently only one instance type.

Instance type – The instance type to be supported by the host. If you do not configure the host to support multiple instance types, you must choose the specific instance type to be supported. Here, you select mac1.metal.

Availability zone – The Availability Zone in which to allocate the host; you can choose any of the availability zones listed, depending on the datacenter you selected.

Instance auto-placement – Enable auto-placement to allow the host to accept untargeted instance launches that match its host configuration. Disable auto-placement to allow the host to accept targeted instance launches only. By default, it will not be enabled.

Host recovery – Host recovery uses host-level health checks to assess the host’s availability and to detect underlying system failures. If failures are detected on the host, host recovery automatically restarts the running instances on the failed host to a new replacement host. By default, it will not be enabled and is not supported for Instance type mac1.metal.

Quantity – The number of dedicated hosts you would like to purchase for this configuration.

The quantity you select here cannot be greater than your quota limit.

Figure 4-6

Dedicated Host settings

Some availability zones might not have the capacity to provision the dedicated host, which will lead to allocation failure; try the allocation again by selecting different availability zones.

After allocation, it becomes available immediately to launch an instance onto it. Figure 4-7 shows how to initiate the provisioning of an EC2 Mac Instance on the dedicated host.

Figure 4-7

Launching Mac Instance on dedicated host

This action will start an EC2 instance creation process. As seen in Figure 4-8, you select the macOS Amazon Machine Image (AMI) to use in launching the Mac EC2 instance.

Figure 4-8

Selecting macOS Amazon Machine Image (AMI) for instance launch

The next step is to select the instance type. The non-grayed option(s) is the available instance type(s) for the selected AMI. In Figure 4-9, the mac1.metal instance type is already selected.

Figure 4-9

Selecting macOS instance type

Instance details can be configured as shown in Figure 4-10. Here, you configure the VPC, subnet, host configuration, etc.

IAM (identity access management) role is not required for this launch, so select None for the IAM role option and continue.

Figure 4-10

Configuring EC2 Mac details

For the instance storage, the default storage can be configured for now as shown in Figure 4-11. The default storage is sufficient for initial deployment and connection.

Figure 4-11

Default Mac instance storage

To allow inbound network connectivity to the Mac instance after launch, security group rules are defined to allow connection to the applicable ports from a specific IP. As shown in Figure 4-12, connections are allowed from my IP address to TCP port 22, which is the secure shell (SSH) protocol port, and TCP port 5900, which is the virtual network computing (VNC) protocol port.

Figure 4-12

Configuring security group rules for inbound connectivity

After security group rules are configured, review all the configuration details entered for accuracy and proceed with the instance launch.

Additionally, for inbound user SSH connectivity to the Mac instance, an EC2 key pair is required. You can use an existing key pair you have created in your account or create a new one. Figure 4-13 shows the process of creating a new key pair before launching the instance.

Figure 4-13

Creating new EC2 key pair before launching the instance

Once the instance is launched, it appears under the Running instances tab under Dedicated Hosts, as shown in Figure 4-14.

Figure 4-14

Dedicated host running instances

You can also view more details about the running instance by selecting it. This redirects you to the EC2 Instances console, as shown in Figure 4-15.

Figure 4-15

Mac EC2 instance details

Deploying with AWS CLI

The EC2 Mac instance can also be provisioned from the AWS command line interface (CLI) . To use the AWS CLI, you must first install it on your workstation and configure your workstation with appropriate identity access management (IAM) credentials.

Information on how to install and configure AWS CLI can be found at https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html.

To provision an EC2 Mac instance, first allocate a dedicated host, as shown in Listing 4-1.

aws ec2 allocate-hosts --region us-east-1 --instance-type mac1.metal --availability-zone us-east-1c --auto-placement "on" --quantity 1

Listing 4-1

Allocating dedicated host

Once you have allocated a dedicated host, you can then create a Mac instance as shown in Listing 4-2. You get a response that contains the Instance ID which you can use for subsequent CLI commands.

aws ec2 run-instances --region us-east-1 --instance-type mac1.metal --placement Tenancy=host --image-id ami-059ff882c04ebed21 --key-name my-key-pair

Listing 4-2

Provisioning EC2 Mac instance on the dedicated host

Verify the instance created by describing its status as shown in Listing 4-3. Use the Instance ID retrieved from the CLI response during the creation.

aws ec2 describe-instance-status --instance-ids <Instance ID>

Listing 4-3

Verifying Instance creation status

Deploying with AWS CloudFormation

AWS CloudFormation is an infrastructure as code service that allows you to model the resources to be deployed in a jSON or YAML template. The template is then passed to AWS CloudFormation to provision the defined resources.

To deploy EC2 Mac instance with CloudFormation, you will define all the components required in a template, as shown in Listing 4-4.

Parameters:

  VpcId:

    Type: AWS::EC2::VPC::Id

    Description: VPC to launch the EC2 Mac Instance

  SubnetId:

    Type: AWS::EC2::Subnet::Id

    Description: |

      Subnet to launch the EC2 Mac Instance............

  MacEc2Host:

    Type: String

    Default: h-xxxxxxxxxxxxxx

    Description: The Host ID of the Allocated.........

  SourceIp:

    Type: String

    Default: 1.2.3.4/32

    Description: |

      Source IP address and subnet mask...............

  Ec2KeyPairName:

    Type: AWS::EC2::KeyPair::KeyName

    Description: Already existing EC2 keypair............

  MacOSAmi:

    Type: AWS::EC2::Image::Id

    Default: ami-059ff882c04ebed21

    Description:latest AMI ID for the Mac OS instance

Resources:

  Ec2MacSecurityGroup:

    Type: AWS::EC2::SecurityGroup

    Properties:

      VpcId: !Ref VpcId

      GroupDescription: Enable SSH and VNC access.........

      SecurityGroupIngress:

      - CidrIp: !Ref SourceIp

        FromPort: 22

        IpProtocol: tcp

        ToPort: 22

      - CidrIp: !Ref SourceIp

        FromPort: 5900

        IpProtocol: tcp

        ToPort: 5900

  MacEc2Instance:

    Type: AWS::EC2::Instance

    Properties:

      Affinity: host

      ImageId: !Ref MacOSAmi

      KeyName: !Ref Ec2KeyPairName

      InstanceType: mac1.metal

      HostId: !Ref MacEc2Host

      SubnetId: !Ref SubnetId

      SecurityGroupIds:

        - !Ref Ec2MacSecurityGroup

      BlockDeviceMappings:

        - DeviceName: /dev/sda1

          Ebs:

            VolumeType: gp2

            VolumeSize: '200'

            DeleteOnTermination: 'true'

Listing 4-4

AWS CloudFormation template for EC2 Mac

The Parameters section of the template contains the input runtime parameters that users will provide, which is then used in the Resources section for resource creation.

As seen in Listing 4-4, two resources defined in the template are the EC2 Mac Security group and the Mac instances. To launch the CloudFormation template, create a stack from your AWS CloudFormation console, select the file, and click next, as shown in Figure 4-16.

Figure 4-16

Selecting the EC2 Mac Cloudformation template for stack creation

Next, you will input the user parameters defined in the template, as shown in Figure 4-17. Some example values are already entered as default values, these should be overridden with information specific to your environment and AWS account.

Figure 4-17

Entering CloudFormation stack parameters

To inspect the resources that will be created before creation, you can create a change set. A change set is shown in Figure 4-18.

Figure 4-18

CloudFormation change set for EC2 Mac creation

Deploying with Terraform

Terraform is an open-source infrastructure as code software tool that allows you to model resource to be provisioned in a template. Terraform was created by HashiCorp and it uses a declarative configuration language known as HashiCorp Configuration Language (HCL) with a .tf file extension. It also supports declaring resources in JSON.

Listing 4-5 shows the definition of the EC2 Mac resources in a terraform HCL template.

provider "aws" {

    region = "us-east-1"

}

variable "awsprops" {

    type = map(string)

    default = {

        vpcid = "vpc-937e84ee"

        subnetid = "subnet-c10c6fcf"

        ami = "ami-059ff882c04ebed21"

        keyname = "my-key-pair"

        hostid = "h-xxxxxxxxxxxx"

        sourceip = "1.2.3.4/32"

    }

}

resource "aws_security_group" "ec2-mac-instance-sg" {

  description = "Enable SSH and VNC access to Mac Instance"

  vpc_id = lookup(var.awsprops, "vpcid")

  // To Allow SSH Transport

  ingress {

    from_port = 22

    protocol = "tcp"

    to_port = 22

    cidr_blocks = [lookup(var.awsprops, "sourceip")]

  }

  // To Allow Port VNC Transport

  ingress {

    from_port = 5900

    protocol = "tcp"

    to_port = 5900

    cidr_blocks = [lookup(var.awsprops, "sourceip")]

  }

}

resource "aws_instance" "ec2-mac-instance" {

  ami = lookup(var.awsprops, "ami")

  key_name = lookup(var.awsprops, "keyname")

  host_id = lookup(var.awsprops, "hostid")

  instance_type = "mac1.metal"

  subnet_id = lookup(var.awsprops, "subnetid")

  vpc_security_group_ids = [

      aws_security_group.ec2-mac-instance-sg.id

  ]

  root_block_device {

    delete_on_termination = true

    volume_type = "gp2"

    volume_size = "200"

  }

}

Listing 4-5

Terraform definition for EC2 Mac resources

Like AWS CloudFormation, the terraform template contains input variables that need to be customized and the resources section where an EC2 security and EC2 Mac instance resources are defined.

To use Terraform, the software must be installed on your workstation and the identity access management (IAM) credentials for the target AWS account must be configured.

See https://learn.hashicorp.com/tutorials/terraform/install-cli for more information on installing Terraform.

To deploy resources defined in a Terraform .tf file, initialize your working directory as shown in Listing 4-6 to make it a Terraform working directory and for all dependencies to be installed.

terraform> terraform init

Initializing the backend...

Initializing provider plugins...

- Finding latest version of hashicorp/aws...

- Installing hashicorp/aws v3.49.0...

- Installed hashicorp/aws v3.49.0 (signed by HashiCorp)

Listing 4-6

Initializing Terraform

You can also validate your Terraform file as shown in Listing 4-7 before attempting to deploy to fix any syntax issue.

terraform> terraform validate

Success! The configuration is valid.

Listing 4-7

Validating Terraform files in directory

To preview all the resources that will be provisioned like a CloudFormation change set, you can execute a terraform plan command. This shows you details about resources to be provisioned and their relationships. Listing 4-8 shows a snippet of the output of this command.

terraform> terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the

following symbols:

  + create

Terraform will perform the following actions:

  # aws_instance.ec2-mac-instance will be created

  + resource "aws_instance" "ec2-mac-instance" {

      + ami                           = "ami-059ff882c04ebed21"

      + arn                           = (known after apply)

      + associate_public_ip_address   = (known after apply)

.

.

.

.

Plan: 2 to add, 0 to change, 0 to destroy.

-----------------------------------------------------------------------------------------------

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

Listing 4-8

Viewing Terrform plan before resource creation

After reviewing all the resources to be created, to start the creation, you apply the configuration shown in Listing 4-9.

terraform> terraform apply

Listing 4-9

Applying terraform configuration

Now that I have shown how to deploy an EC2 Mac server, next we will explore how you can connect to it.

Connecting to the Amazon EC2 Mac Server

For every new EC2 Mac server, password authentication is disabled by default. The default access method for the first time is using SSH; the ec2-user account is configured for SSH remote login. To connect using SSH, the instance must have been launched using a key pair and its security group allows SSH access from the workstation where the SSH connection will originate.

After connecting with SSH, password authentication can be configured to allow connection using Virtual Network Computing (VNC) clients.

Connecting with SSH

To connect with SSH, there are various options discussed in this section to suit your operating system and/or preference.

Command Terminal on Linux or Windows

The most common method of connecting on both Windows and Linux operating system is via the command terminal and using the operating in-built SSH client. The command to SSH using the command terminal is shown in the example in Listing 4-10.

ssh -i /path/my-key-pair.pem ec2-user@ec2-12-23-45-67.compute-1.amazonaws.com

Listing 4-10

SSH on command terminal

If you get an error about your key being too open and unprotected, change the key’s permission with chmod 400 /path/my-key-pair.pem.

Putty on Windows

Putty is a free and open-source telnet and SSH client for windows that can also be used to connect to the EC2 Mac instance. To use Putty, you must convert the .pem key pair to a .ppk key for compatibility. After conversion, you provide the .ppk key as shown in Figure 4-19.

See https://aws.amazon.com/premiumsupport/knowledge-center/ec2-ppk-pem-conversion/ for more information on converting a pem key to ppk format.

Figure 4-19

Providing authentication for Putty session

With authentication configured, you specify the hostname in the format ec2-user@<host name> and the port and connection type, as shown in Figure 4-20.

Figure 4-20

Starting Putty SSH session

Once you click Open, you are logged in to the command terminal of the EC2 Mac server, as shown in Figure 4-21.

Figure 4-21

Command terminal of EC2 Mac Server

Connecting with VNC Client

To connect to the user interface (UI) of the Mac server, you must use a VNC client. But before using a VNC client, there are a couple of prerequisites that need to be configured.

Setting Up Login Password

By default, the ec2-user has password authentication disabled. To use a VNC client and connect to the UI, password must be created for the ec2-user as shown in Listing 4-11.

ec2-user@ip-172-31-47-239 ~ % sudo passwd ec2-user

Changing password for ec2-user.

New password:

Retype new password:

################################### WARNING ###################################

# This tool does not update the login keychain password.                      #

# To update it, run `security set-keychain-password` as the user in question, #

# or as root providing a path to such user's login keychain.                  #

###############################################################################

Listing 4-11

Setting up password for default ec2-user account

Keep the password you set here safe as it will be used for logging in to your server using VNC clients.

Setting Up Apple Remote Desktop

To enable remote desktop connection on the Mac server, the Apple remote desktop agent must be started from the terminal of the Mac server as shown in Listing 4-12.

ec2-user@ip-172-31-47-239 ~ % sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart \

-activate -configure -access -on \

-restart -agent -privs -all

Starting...

Warning: macos 10.14 and later only allows control if Screen Sharing is enabled through System Preferences.

Activated Remote Management.

Stopped ARD Agent.

ec2-user: Set user remote control privileges.

ec2-user: Set user remote access.

Done.

Listing 4-12

Starting Apple Remote Desktop Agent

Setting Up SSH Port Forwarding

You can connect directly to the IP address of the Mac server on port 5900 (VNC Port) from the VNC client on your workstation, but the VNC protocol is not considered secure. A more secure approach is to tunnel VNC traffic over SSH using SSH port forwarding feature from your workstation.

Listing 4-13 shows how you can forward all traffic to your localhost port 5900 to the EC2 Mac server port 5900. Hence, to connect to the Mac server, you send traffic to localhost port 5900.

ssh -L 5900:localhost:5900 -i /path/my-key-pair.pem ec2-user@your-instance-public-dns-name

Listing 4-13

SSH tunneling VNC traffic

Downloading VNC Client and Connect

There are various VNC client options available for use depending on your operating system. One such option is VNC Viewer, which is available across multiple operating systems; it can be downloaded as shown in Figure 4-22.

Figure 4-22

Downloading VNC client

It can be installed on a Windows operating system, as shown in Figure 4-23.

Figure 4-23

Installing VNC client on Windows

After installation, ensure that port 5900 SSH tunnelling is set up as described in Listing 4-13, then you can start connection to the Mac server by connecting to your localhost, as shown in Figure 4-24.

Figure 4-24

Connecting to EC2 Mac server with VNC client

After successful connection, you would see the login prompt of your EC2 Mac server, as shown in Figure 4-25.

Figure 4-25

Login prompt for the EC2 Mac server

In Listing 4-11, I showed you how to set up password for the default ec2-user. The password you set up there is what the ec2-user will use to log in.

After successful authentication, you will be logged on, as shown in Figure 4-26.

Figure 4-26

EC2 Mac server home screen

Monitoring EC2 Mac Server with Amazon CloudWatch

By default, EC2 Mac instance emits metrics to Amazon CloudWatch for performance monitoring. It sends metrics such as CPU utilization, network usage information, etc. The metrics of an EC2 Mac instance can be viewed on the EC2 Console by selecting on the instance and going to the Monitoring tab. An example of this is shown in Figure 4-27.

Figure 4-27

Instance metrics information on the EC2 console

The metrics can also be viewed on the Amazon CloudWatch console. The CloudWatch console provides you with more graph options and allows you to compare multiple metrics on the same graph, as shown in Figure 4-28.

Figure 4-28

Instance metrics graph on Cloudwatch

To disable the agent that sends system monitoring metrics such as CPU utilization to CloudWatch, you can run the setup as shown in Listing 4-14.

$ sudo setup-ec2monitoring [enable | disable]

Listing 4-14

Configuring EC2 system monitoring for Mac instance

Cleaning Up Amazon EC2 Mac Server

To clean up an Amazon EC2 server if you’re not actively using it, you terminate the EC2 instance first and then release the dedicated host.

To terminate the instance, from the EC2 dashboard, choose Instances and select the mac1.metal instance you want to terminate. Choose Instance state and then select Terminate Instance .

As Amazon EC2 Mac instances are not charged by instance but by dedicated hosts, terminating the instance doesn’t stop the charges, the dedicated host must be released to stop incurring charges.

You cannot release the dedicated host until after 24 hours has passed since allocating the host.

To release the host, from the EC2 dashboard, choose Dedicated hosts and select the dedicated host you want to release. Choose Actions and then select Release host.

Summary

We covered the details around deploying an EC2 macOS instance by exploring different methods of deployment. I also showed you the different connectivity options while also covering the prerequisites for connecting to an EC2 Mac server.

In the next chapter, we will build on this, and I will show you how to install and set up various iOS development and build tools on your EC2 macOS instance.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

A. OlaoyeBeginning DevOps on AWS for iOS Developmenthttps://doi.org/10.1007/978-1-4842-8023-2_5

5. Setting Up Development Tools on macOS Instance

Abdullahi Olaoye1  

(1)

McKinney, TX, USA

To start development on an Amazon EC2 Mac instance, just like any other Mac build server, iOS build tools such as Xcode must be installed. There are other open-source and third-party automation tools such as Jenkins, Fastlane, etc., which integrate with Xcode to add automation and improve iOS development processes.

In this chapter, we will explore how to install Xcode, Jenkins, Fastlane, and GitLab Runner on an Amazon EC2 macOS instance. But, before installing software, there must be adequate storage space so we will start by seeing how the volume size of a macOS instance can be increased.

Increasing macOS Instance Volume Size

To install an iOS application build tool like Xcode, sufficient disk space must be available. The macOS instance volume size can be modified at launch or after the Mac instance is running. If the volume size is not modified at launch time, the default disk size that is provisioned with the Amazon EC2 Mac instance might not be sufficient.

The macOS instance uses an Amazon Elastic Block Storage (EBS) volume. Increasing the storage capacity of a running macOS instance involves increasing the capacity of the attached EBS volume, rebooting the instance, and making the increased disk space available for use.

The instance volume size can be seen by selecting the macOS instance on the EC2 console as shown in Figure 5-1. To increase the volume size, click on the EBS volume ID as shown in Figure 5-1; this will direct you to the EBS console.

Figure 5-1

Volume size shown in macOS instance details

Access the volume modification page as shown in Figure 5-2.

Figure 5-2

Selecting Modify Volume option

Modify the instance EBS volume size. The example shown in Figure 5-3 increases the volume to 200 GB.

Figure 5-3

Modifying volume to the desired size

After resizing the attached EBS volume of a running macOS instance, it will not be visible within the instance until a reboot occurs. You can reboot the macOS instance from the EC2 console as shown in Figures 5-4 and 5-5.

Figure 5-4

Rebooting macOS instance after volume resizing

Figure 5-5

Confirming instance reboot

After a macOS reboot following an EBS volume resizing, the new disk size should be visible on the macOS instance, as shown in Listing 5-1.

ec2-user@ip-172-31-47-239 ~ % diskutil list external physical

/dev/disk1 (external, physical):

   #:           TYPE NAME            SIZE       IDENTIFIER

   0:      GUID_partition_scheme   *214.7 GB      disk1

   1:            EFI EFI            209.7 MB      disk1s1

   2:   Apple_APFS Container disk2   64.2 GB      disk1s2

Listing 5-1

Listing disk size

As shown in Listing 5-1, the new disk size is visible, but the APFS container size is still smaller, hence the increased disk space is not being used yet. The added disk size can be made usable as shown in Listing 5-2.

ec2-user@ip-172-31-47-239 ~ % PDISK=$(diskutil list physical external | head -n1 | cut -d" " -f1)

ec2-user@ip-172-31-47-239 ~ % APFSCONT=$(diskutil list physical external | grep "Apple_APFS" | tr -s " " | cut -d" " -f8)

ec2-user@ip-172-31-47-239 ~ % yes | sudo diskutil repairDisk $PDISK

Repairing the partition map might erase disk1s1, proceed? (y/N) Started partition map repair on disk1

Checking prerequisites

Listing 5-2

Repairing disk and making increased size usable

After repairing, the last step is to resize the Apple APFS container as shown in Listing 5-3.

ec2-user@ip-172-31-47-239 ~ % sudo diskutil apfs resizeContainer $APFSCONT 0

Started APFS operation

Aligning grow delta to 150,323,855,360 bytes and targeting a new physical store size of 214,538,608,640 bytes

Determined the maximum size for the targeted physical store of this APFS Container to be 214,537,580,544 bytes

Listing 5-3

Resizing APFS container size

Setting Up Xcode

Xcode can be installed manually on the macOS instance from the App Store just as discussed in Chapter 1. After Xcode installation, to begin using Xcode, you must accept the Xcode and iOS SDK License Agreement, as shown in Figure 5-6.

Figure 5-6

Xcode and iOS SDK License Agreement

If prompted to enter password to grant Xcode permissions, enter your user password. If using the default ec2-user, this is the user password that is created for the user as discussed in Chapter 4.

Once Xcode is ready for usage, it should launch into its welcome page, as shown in Figure 5-7.

Figure 5-7

Xcode welcome page

Xcode Command Line Tools

When Xcode is installed from the App Store, its command line tools are also installed without any further action. The installation can be verified from the command terminal, as shown in Listing 5-4.

ec2-user@ip-172-31-47-239 ~ % xcode-select --install

xcode-select: error: command line tools are already installed, use "Software Update" to install updates

ec2-user@ip-172-31-47-239 ~ % xcodebuild

Command line invocation:

    /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild

User defaults from command line:

    IDEPackageSupportUseBuiltinSCM = YES

xcodebuild: error: The directory /Users/ec2-user does not contain an Xcode project .

ec2-user@ip-172-31-47-239 ~ % xcodebuild -version

Xcode 12.5.1

Build version 12E507

Listing 5-4

Verifying Xcode command line tools installation

Setting Up Jenkins

Jenkins is an open-source automation and application build tool. It integrates with a variety of other tools, which makes it suitable for various types of builds and automations. Jenkins supports distributed builds; you can use different build environments for each build project using agents that run jobs in parallel.

These agents are controlled by a controller, which is the main Jenkins node where all build jobs are configured and scheduled. The agents can be the same type of environment as the controller where jobs are scheduled to balance workload or a different environment from the main controller where jobs are scheduled on the agent based on the build environment required by the application being built.

Let us explore in the next two sections how you can configure a Jenkins controller in a Linux environment on an Amazon EC2 instance and have a macOS build agent for building iOS applications.

Provisioning Jenkins Controller

The controller will be provisioned on an Amazon EC2 Linux instance, so, firstly, an EC2 instance must be created from the EC2 console, as shown in Figure 5-8.

Figure 5-8

Starting Amazon EC2 instance launch from EC2 Console

The latest Amazon Linux Amazon Machine Image (AMI) for the applicable AWS region is selected, as shown in Figure 5-9.

Figure 5-9

Selecting latest Amazon Linux AMI

Select the instance type and configure the instance and storage capacity as desired. For the security group, SSH (TCP Port 22) access is needed to enable access into the Jenkins server. Jenkins also serves traffic on TCP port 8080 so this must be allowed in the security group. An example security group configuration is shown in Figure 5-10. These rules can be further restricted with the source IP address of your workstation.

Figure 5-10

Configuring security group rules

After the instance launch, the next step is to log in to the instance through SSH to install the Jenkins software.

Since an Amazon Linux EC2 instance is provisioned, yum package manager will be used to install Jenkins. Similarly, package managers in other operating systems can be used to install Jenkins on those operating systems.

First, a quick software update of all packages on the instance is performed, as in Listing 5-5.

[ec2-user ~]$ sudo yum update

Listing 5-5

Updating software packages

After updating software packages, the Jenkins repo can be added to yum, as shown in Listing 5-6.

[ec2-user ~]$ sudo wget -O /etc/yum.repos.d/jenkins.repo \

    https://pkg.jenkins.io/redhat-stable/jenkins.repo

Listing 5-6

Adding Jenkins to yum repo

Next is to import the key file for RPM to verify the package. This is shown in Listing 5-7.

[ec2-user ~]$ sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key

[ec2-user ~]$ sudo yum upgrade

Listing 5-7

Import rpm key file to verify package

Jenkins can then be installed, as shown in Listing 5-8.

[ec2-user ~]$ sudo yum install jenkins java-1.8.0-openjdk-devel -y

[ec2-user ~]$ sudo systemctl daemon-reload

Listing 5-8

Installing Jenkins

After installation, Jenkins can be started as a service, as shown in Listing 5-9.

[ec2-user ~]$ sudo systemctl start jenkins

Listing 5-9

Starting Jenkins as a service

Jenkins status can be checked as shown in Listing 5-10.

[ec2-user ~]$ sudo systemctl status jenkins

Listing 5-10

Checking Jenkins status

To set up Jenkins initially, it requires an initial administrator password. This password can be retrieved as shown in Listing 5-11.

[ec2-user ~]$ sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Listing 5-11

Retriving initial Admin Password

To access your Jenkins server for the first time through its management interface, connect to http://<your_server_public_DNS>:8080 from your web browser. The initial setup page is as shown in Figure 5-11. Enter the initial password retrieved as shown in Listing 5-11.

Figure 5-11

Jenkins initial setup page

To support different types of applications and tools, Jenkins uses plugins to extend its functionalities. During Jenkins initial setup, you can select specific plugins to install, or install the suggested plugins as shown in Figure 5-12.

Figure 5-12

Installing Jenkins plugin

An initial admin user must also be created to authenticate into Jenkins. The initial admin user can be set up as shown in Figure 5-13.

Figure 5-13

Creating initial admin user

After all the plugins are installed and the initial admin user is set up, Jenkins is ready for use to start creating jobs and registering build agents. A fresh Jenkins installation welcome page is as shown in Figure 5-14.

Figure 5-14

Jenkins welcome page

EC2 Mac Instance As Jenkins Build Agent

After creating the Jenkins controller, you can set up build agents for distributed builds. To build iOS applications, a macOS build agent must be present. Let us explore how to add an EC2 Mac instance as a build agent to the Jenkins controller via SSH.

To add a Jenkins build agent, Java must be installed. Listing 5-12 shows how to install Java development kit with Homebrew on EC2 Mac.

ec2-user@ip-172-31-47-239 ~ % su ec2-user -c '/usr/local/bin/brew install openjdk@8'

Password:

Updating Homebrew...

==> Auto-updated Homebrew!

Updated 1 tap (homebrew/core).

==> Updated Formulae

Updated 20 formulae.

==> Downloading https://ghcr.io/v2/homebrew/core/libpng/manifests/1.6.37

######################################################################## 100.0%

.

.

.

.

.

==> Installing dependencies for openjdk@8: libpng and freetype

==> Installing openjdk@8 dependency: libpng

Listing 5-12

Installing OpenJDK on EC2 Mac

After installing OpenJDK with Homebrew, you create a symbolic link for it and verify installation as shown in Listing 5-13.

sh-3.2# ln -sfn /usr/local/opt/openjdk@8/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-8.jdk

sh-3.2# java -version

openjdk version "1.8.0_282"

OpenJDK Runtime Environment (build 1.8.0_282-bre_2021_01_20_16_06-b00)

OpenJDK 64-Bit Server VM (build 25.282-b00, mixed mode)

Listing 5-13

Create symlink for OpenJDK and verify Installation

After having Java installed, the next requirement is to create a Jenkins user and set up SSH access for that user. The Jenkins controller will use this user to connect to the macOS build agent via SSH to submit build jobs. You can create a new Jenkins user as shown in Listing 5-14.

sh-3.2# sysadminctl -addUser jenkins

2021-07-10 19:50:42.793 sysadminctl[6437:155160] --------------

2021-07-10 19:50:42.793 sysadminctl[6437:155160] No clear text password or interactive option was specified (adduser, change/reset password will not allow user to use FDE) !

2021-07-10 19:50:42.793 sysadminctl[6437:155160] --------------

2021-07-10 19:50:42.970 sysadminctl[6437:155160] Creating user record...

2021-07-10 19:50:43.043 sysadminctl[6437:155160] Assigning UID: 503 GID: 20

2021-07-10 19:50:43.076 sysadminctl[6437:155160] Creating home directory at /Users/jenkins

Listing 5-14

Creating Jenkins user

With the user created, you can either set up a new key or reuse the existing default ec2-user key on the macOS instance. Listing 5-15 shows how to reuse the default ec2-user SSH credentials with the new Jenkins user.

sh-3.2# mkdir /Users/jenkins/.ssh/

sh-3.2# cp -a /Users/ec2-user/.ssh/. /Users/jenkins/.ssh/

sh-3.2# cp /Users/ec2-user/.ssh/* /Users/jenkins/.ssh/.

sh-3.2# chown -R jenkins /Users/jenkins/.ssh

sh-3.2# chmod 700 /Users/jenkins/.ssh

sh-3.2# chmod 600 /Users/jenkins/.ssh/authorized_keys

Listing 5-15

Setting up SSH keys for Jenkins user

Next let us ensure that the Jenkins controller has SSH access to the EC2 Mac instance from the security groups.

As shown in Figure 5-15, TCP port 22 access is configured for the /32 private IP address of the Jenkins controller instance. The private IP address is used because the Jenkins controller and EC2 Mac instance are in the same Virtual Private Cloud (VPC). If they are in different VPCs and the Jenkins controller will connect with the Mac instance via the Internet, you will add the public IP address of the Jenkins controller instead.

Figure 5-15

Security group configuration for EC2 Mac instance

With Java installed, Jenkins user created, SSH access set up, and security group configured, you are ready to add the EC2 macOS instance as a build node.

On the Jenkins user interface, select “Manage Jenkins” option, then select “Manage Nodes and Clouds” as shown in Figure 5-16.

Figure 5-16

Managing Jenkins options

Initiate the process to add a new node and provide a name for the node, as shown in Figure 5-17.

Figure 5-17

Adding new Jenkins node

Connectivity information of the node will be configured which includes the IP address of the node, the launch method, label, credentials of the node, etc. The label is configured in build jobs and used by the controller to identify the build agent to schedule a specific job on. An example configuration of a node with SSH launch method and an iOS label is shown in Figure 5-18.

The SSH credentials must be configured separately. The credentials can be created in advance or at the time of adding a node. Figure 5-18 shows how to add new credentials when configuring the build node.

Figure 5-18

Configuring new credentials for Jenkins node

To add SSH credentials, you provide the Jenkins username and the SSH private key. To get the private key content, open the private key file that is configured for Jenkins user SSH access.

In this example, the same SSH private key is used for ec2-user and Jenkins user as configured in Listing 5-15 so retrieve the content of that private key from your workstation.

Figure 5-19 shows an example of configuring SSH credentials.

Figure 5-19

Configuring SSH credentials in Jenkins

Environment variables such as path variables can be configured for the node, as shown in Figure 5-20.

Figure 5-20

Configuring environment variables for the node

Save the configuration for the new node. From Figure 5-18, the host key verification is configured as a manual trust, so after the new node configuration is saved, the Jenkins controller attempts to connect via SSH to the build agent and a prompt is shown on the left-side options of the Jenkins node with a request to trust the Jenkins controller; select this to allow the connection to proceed.

You can view the console output on Jenkins to view the status and see it in action. A console output example for a successful agent added is shown in Listing 5-16.

[07/11/21 17:31:16] [SSH] Checking java version of java

[07/11/21 17:31:16] [SSH] java -version returned 1.8.0_282.

[07/11/21 17:31:16] [SSH] Starting sftp client.

[07/11/21 17:31:16] [SSH] Copying latest remoting.jar...

[07/11/21 17:31:16] [SSH] Copied 1,502,119 bytes.

Expanded the channel window size to 4MB

[07/11/21 17:31:16] [SSH] Starting agent process: cd "/Users/Jenkins" && java  -jar remoting.jar -workDir /Users/Jenkins -jar-cache /Users/Jenkins/remoting/jarCache

Jul 11, 2021 5:31:17 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir

INFO: Using /Users/Jenkins/remoting as a remoting work directory

Jul 11, 2021 5:31:17 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging

INFO: Both error and output logs will be printed to /Users/Jenkins/remoting

<===[JENKINS REMOTING CAPACITY]===>channel started

Remoting version: 4.7

This is a Unix agent

Evacuated stdout

Agent successfully connected and online

Listing 5-16

Output snippet of adding Jenkins agent

After a build node is added, it will be visible in the list of nodes, as shown in Figure 5-21.

Figure 5-21

Jenkins build nodes

Setting Up Fastlane

Fastlane is an open-source automation tool that is used to automate mobile application development. It automates all aspects of building an iOS application from development to release to the App Store. Its major features are

Automate screenshot for use in App Store

Automate beta deployment

App Store deployment

Code signing

Fastlane can be integrated with Jenkins to create a full end-to-end Continuous Integration Continuous Delivery (CICD) process for iOS application development.

Fastlane can simply be installed on a macOS workstation using Homebrew, as shown in Listing 5-17.

ec2-user@ip-172-31-47-239 ~ % brew install fastlane

Updating Homebrew...

==> Auto-updated Homebrew!

Updated 2 taps (homebrew/core and homebrew/cask).

==> Updated Formulae

Updated 52 formulae.

==> Updated Casks

Updated 13 casks.

.

.

.

.

.

==> Installing fastlane

==> Pouring fastlane--2.187.0.big_sur.bottle.1.tar.gz

Listing 5-17

Fastlane installation output snippet

After installation, you can verify installation and start using Fastlane. An example of starting Fastlane and an example output are shown in Listing 5-18.

Listing 5-18. Starting Fastlane

Setting Up GitLab

GitLab is a DevOps platform that provides a host of features to increase agility in software development process. Some of the features are git repository manager for managing source code, continuous integration, and deployment pipelines for automatically building and deploying applications etc.

For its continuous integration and deployment feature, it uses GitLab runner. GitLab Runner is an application that works with GitLab CICD to run jobs in a pipeline. The GitLab Runner application can be installed on a separate infrastructure and operating system depending on the job the runner will perform.

GitLab Runner can also be installed on macOS for running iOS-related jobs, so let us explore how to set up GitLab and install a GitLab Runner on macOS EC2 instance.

Creating GitLab Project

To start using GitLab, you must sign up for a GitLab account. You can sign up for free on https://gitlab.com/. Then, create a project as shown in Figure 5-22.

Figure 5-22

Create GitLab project

Next, you are given the option to choose how you want to create the project, as shown in Figure 5-23, you can create a blank project or use a template which will prepopulate your project with files.

Figure 5-23

Selecting project initial configuration

Configure the project name, URL, and the visibility level (Private or Public). Figure 5-24 shows an example project configuration.

Figure 5-24

Configuring and creating the GitLab project

Setting Up macOS GitLab Runner

When you create a GitLab project, a repository is created for the project, and you get access to lots of DevOps features, among which is CICD. To set up a GitLab CICD pipeline that can build iOS applications, a macOS GitLab Runner must be set up.

GitLab Runner configuration for a project can be accessed in the project settings, as shown in Figure 5-25.

Figure 5-25

GitLab project CICD settings

To register a GitLab runner on your macOS instance, it requires the URL of the GitLab location and a unique registration token. You can retrieve both by expanding Runners in project CICD settings, as shown in Figure 5-26.

Figure 5-26

GitLab Runner registration token and URL

The GitLab Runner is available as a binary. You can download the binary and provide appropriate permission on your macOS instance, as shown in Listing 5-19.

ec2-user@ip-172-31-47-239 ~ % sudo curl --output /usr/local/bin/gitlab-runner "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-darwin-amd64"

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100 53.0M  100 53.0M    0     0  92.6M      0 --:--:-- --:--:-- --:--:-- 92.4M

ec2-user@ip-172-31-47-239 ~ % sudo chmod +x /usr/local/bin/gitlab-runner

Listing 5-19

Download GitLab Runner binary for macOS

Now you can register the runner as shown in Listing 5-20. The registration is an interactive operation where you will provide the registration token, GitLab URL, runner description, etc.

ec2-user@ip-172-31-47-239 ~ % gitlab-runner register

Runtime platform                                    arch=amd64 os=darwin pid=4212 revision=c1edb478 version=14.0.1

WARNING: Running in user-mode.

WARNING: Use sudo for system-mode:

WARNING: $ sudo gitlab-runner...

Enter the GitLab instance URL (for example, https://gitlab.com/):

https://gitlab.com/

Enter the registration token:

nxxxxxxxxxxxxxxi

Enter a description for the runner:

[ip-172-31-47-239.ec2.internal]: mac os runner

Enter tags for the runner (comma-separated):

Registering runner... succeeded                     runner=nYR2beCG

Enter an executor: custom, docker, parallels, docker-ssh, shell, ssh, virtualbox, docker+machine, docker-ssh+machine, kubernetes:

shell

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Listing 5-20

GitLab Runner registration

A successful runner registration does not install the runner, you must install the runner separately. Listing 5-21 shows how to install and start the GitLab Runner.

ec2-user@ip-172-31-47-239 ~ % gitlab-runner install

Runtime platform                                    arch=amd64 os=darwin pid=4237 revision=c1edb478 version=14.0.1

ec2-user@ip-172-31-47-239 ~ % gitlab-runner start

Runtime platform                                    arch=amd64 os=darwin pid=4240 revision=c1edb478 version=14.0.1

Listing 5-21

Installing and starting GitLab Runner

Once the GitLab Runner is installed, you can start scheduling jobs to this runner from a GitLab CICD pipeline. To verify that the runner has been registered with your GitLab project and ready to receive jobs, ensure that you can see the registered macOS runner on the list of runners on your GitLab project. Figure 5-27 shows the macOS instance registered as a runner on GitLab.

Figure 5-27

Viewing registered macOS runner on GitLab

See https://docs.gitlab.com/ee/ci/pipelines/index.html for more information on setting up GitLab CICD pipelines.

Summary

Setting up an iOS application development environment requires installation of different software such as iOS-specific Xcode and other third-party and open-source tools. We have covered the installation and setup of all these tools on a macOS Amazon EC2 instance to get it ready for development.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

A. OlaoyeBeginning DevOps on AWS for iOS Developmenthttps://doi.org/10.1007/978-1-4842-8023-2_6

6. Source Control Management with AWS CodeCommit

Abdullahi Olaoye1  

(1)

McKinney, TX, USA

As discussed in Chapter 3, AWS CodeCommit is a managed private Git repository. When interacting and using CodeCommit, you use the standard Git tools, and all the standard Git concepts and features apply.

In this chapter, we will dive deep into how to use AWS CodeCommit for source control management starting with how to connect to a CodeCommit repository to add source code and how features like pull requests, branching, etc., work.

Before diving right into using AWS CodeCommit, let’s cover the basic Git fundamentals as a refresher.

All example code used in this chapter can be found at https://github.com/abdulola/devops-on-aws-ios-development/tree/main/Chapter_6.

Git Fundamentals

Git is an open-source version control system. It tracks changes you make to files in a directory where it is enabled, which allows you to revert to any version of your files at any point in time. It also makes collaboration easy as it allows changes made to files by different people be merged into one.

Git Installation

Git client for all major platforms can be downloaded from the Git website as seen in Figure 6-1.

The download website can be accessed at https://git-scm.com/downloads.

Figure 6-1

Git client download

It can be downloaded from the command line with package manager for different operating systems. Listing 6-1 shows how to install Git for macOS and Debian/Ubuntu systems, respectively.

macOS

$ brew install git

Debian/Ubuntu

$ apt-get install git

Listing 6-1

Git installation with package managers

Initializing a Git Repository

Once Git is installed, to enable Git in a directory, it must be initialized. Initializing Git in a directory means you want it to start version control and start tracking all changes in that directory. Git can be initialized in a directory as seen in Listing 6-2.

$ git init

Initialized empty Git repository in /home/cloudshell-user/git-diretory/.git/

$ git status

On branch master

No commits yet

nothing to commit (create/copy files and use "git add" to track)

Listing 6-2

Initializing Git repository and checking Git status

Recording Changes to Git Repository

As seen in Listing 6-2, there are no files for Git to track in the directory it is initialized in because it’s an empty directory. New files are added to the directory as shown in Listing 6-3.

$ touch file1 file2 file3

$ git status

On branch master

No commits yet

Untracked files :

  (use "git add <file>..." to include in what will be committed)

        file1

        file2

        file3

nothing added to commit but untracked files present (use "git add" to track)

Listing 6-3

Adding new files and checking Git status

Files are added to the directory but not added for Git to start tracking. For Git to start tracking these new files, first the changes should be staged as shown in Listing 6-4.

$ git add .

$ git status

On branch master

No commits yet

Changes to be committed:

  (use "git rm --cached <file>..." to unstage)

        new file:   file1

        new file:   file2

        new file:   file3

Listing 6-4

Staging changes and checking Git status

After changes have been staged, the last step for Git to start tracking these files is to commit the changes. If this is your first time using Git, you must configure your name and email as shown in Listing 6-5. When committing files to Git, a comment must be provided to describe the changes that are being committed; this is provided with the -m flag, as shown in Listing 6-5.

$ git config --global user.name "Enter Your Name"

$ git config --global user.email "Enter Your Email"

$ git commit -m "Add files"

[master (root-commit) 6ebb411] Add files

 3 files changed, 0 insertions(+), 0 deletions(-)

 create mode 100644 file1

 create mode 100644 file2

 create mode 100644 file3

$ git status

On branch master

nothing to commit, working tree clean

Listing 6-5

Committing changes and checking Git status

Cloning and Working with Remote Git Repositories

Sometimes you already have an existing project stored in a remote Git repository such as GitHub, GitLab, AWS CodeCommit, etc., that you want to work on locally. To do this, you download the repository locally by cloning it. After a repository is cloned locally, you can make changes to the files and go through the Git lifecycle described earlier to start tracking your changes.

An example of cloning a repository is shown in Listing 6-6. Here, I cloned the public AWS CLI GitHub repository. To clone a private repository that you own, you will be prompted to authenticate.

$ git clone https://github.com/aws/aws-cli.git

Cloning into 'aws-cli'...

remote: Enumerating objects: 84578, done.

remote: Counting objects: 100% (790/790), done.

remote: Compressing objects: 100% (246/246), done.

remote: Total 84578 (delta 556), reused 763 (delta 543), pack-reused 83788

Receiving objects: 100% (84578/84578), 53.34 MiB | 20.19 MiB/s, done.

Resolving deltas: 100% (57974/57974), done.

Listing 6-6

Cloning remote Git repository

After a remote repository is cloned locally, the local repository is automatically configured to point to the remote repository for future sync operations. For example, if there are updates in the remote repository, the local repository can pull these changes to merge with the local repository, or if there are changes in the local repository, they need to be pushed to update the remote repository.

Listing 6-7 shows how you can see the remote repository configured for a local repository, pull and push operations to keep both local and remote repository in sync, and how you can configure additional remote repositories with a different name for a local repository. The default short name used by Git for a remote repository is origin, and a new remote repository origin2 is configured, as seen in Listing 6-7.

$ cd aws-cli/

$ git remote -v

origin  https://github.com/aws/aws-cli.git (fetch)

origin  https://github.com/aws/aws-cli.git (push)

$ git pull origin

Already up to date.

$ git push origin master

$ git remote add origin2 https://github.com/aws/aws-cli.git

$ git remote -v

origin2 https://github.com/aws/aws-cli.git (fetch)

origin2 https://github.com/aws/aws-cli.git (push)

origin  https://github.com/aws/aws-cli.git (fetch)

origin  https://github.com/aws/aws-cli.git (push)

Listing 6-7

Listing remote repositories and configuring a new remote repository

Git Branch

In Git, a Branch is a pointer to a commit. It is used to separate ongoing work on files and maintain separate concurrent versions of your Git repository. If a file exists on multiple branches, changes made to this file on one branch doesn’t affect the same file in another branch. Branches are typically used to develop new features, store a specific version of your project from a particular commit, etc.

In a new Git repository, when you create your first commit, a default branch is created for you. Additional branches can be created to track different versions of your project.

Pull Requests

Pull requests allow you and other users in a repository to review, add comments, and merge code changes from one branch to another. Pull requests is a way multiple repository users collaborate by reviewing code changes and fixing bugs before a new version of an application is released.

A typical example is when you have a main Git branch which holds the production deployed version of your application and new features are developed in separate feature branches. When a feature from the feature branch is ready for production deployment, a pull request allows team members to review code changes or additions from the new feature before merging the feature branch to the main branch.

Creating an AWS CodeCommit Repository

In Chapter 3, AWS CodeCommit was introduced, and I covered how a CodeCommit repository can be created on the AWS Management Console. In this chapter, we are going to dive deeper to see the various ways a CodeCommit repository can be created, such as AWS Command Line Interface (CLI), CloudFormation, Terraform, and AWS SDK.

Creating with AWS Command Line Interface (CLI)

AWS CLI must first be installed and configured. To start using AWS CLI, it must be configured with credentials that have permission to perform the intended operation. Listing 6-8 shows how you configure AWS CLI and an example output.

It can be installed by referring to the AWS documentation at https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html.

$ aws configure

AWS Access Key ID [None]:

AWS Secret Access Key [None]:

Default region name [None]:

Default output format [None]:

Listing 6-8

Configuring AWS CLI

After configuring your AWS CLI environment, a CodeCommit repository can be created with an example output, as shown in Listing 6-9.

$ aws codecommit create-repository --repository-name devops-ios-repository --repository-description "Example Description"

{

    "repositoryMetadata": {

        "accountId": "123456789101",

        "repositoryId": "abcdedgh-6234-457b-b2c3-184babcdefgh",

        "repositoryName": "devops-ios-repository",

        "repositoryDescription": "Example Description",

        "lastModifiedDate": "2021-08-14T21:20:03.893000+00:00",

        "creationDate": "2021-08-14T21:20:03.893000+00:00",

        "cloneUrlHttp": "https://git-codecommit.us-east-1.amazonaws.com/v1/repos/devops-ios-repository",

        "cloneUrlSsh": "ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/devops-ios-repository",

        "Arn": "arn:aws:codecommit:us-east-1:123456789101:devops-ios-repository"

    }

}

Listing 6-9

Creating a CodeCommit repository with AWS CLI

Creating with AWS CloudFormation

To create a CodeCommit Repository with CloudFormation, the CloudFormation template can be defined as shown in Listing 6-10.

AWSTemplateFormatVersion: "2010-09-09"

Parameters:

  RepositoryName:

    Type: String

    Default: devops-ios-repository

    Description: Name of the CodeCommit Repository

Resources:

  MyRepo:

    Type: AWS::CodeCommit::Repository

    Properties:

      RepositoryName: !Ref RepositoryName

      RepositoryDescription: Example Description

Listing 6-10

CloudFormation template to create CodeCommit repository

As seen in the listing, the CloudFormation template is in YAML format and only contains a CodeCommit resource. The CloudFormation stack can be created with this YAML content, as shown in Figure 6-2.

Figure 6-2

Creating CodeCommit repository with CloudFormation

The repository name is parameterized, as shown in Listing 6-10; a default name is defined in the template, but this can be modified. The parameter can be modified in the stack details page, as shown in Figure 6-3.

Figure 6-3

Configuring name of the CodeCommit repository

Figure 6-4 shows the CloudFormation Stack in CREATE_COMPLETE state which signifies the CodeCommit repository has been created. The resources tab shows the list of resources created by the CloudFormation stack.

Figure 6-4

CodeCommit Cloudformation stack created

The CodeCommit repository created can be seen in the CodeCommit console, as shown in Figure 6-5.

Figure 6-5

CodeCommit repository created

Creating with Terraform

Terraform must first be installed and configured by referring to the official Terraform documentation. The Terraform code for a CodeCommit repository creation can be defined as seen in Listing 6-11.

provider "aws" {

    region = "us-east-1"

}

resource "aws_codecommit_repository" "test" {

  repository_name = "devops-ios-repository"

  description     = "Example Description"

}

Listing 6-11

Terraform code to create CodeCommit repository

The code should be saved in a file with the .tf file extension and saved in a dedicated Terraform directory for the intended operation. In the dedicated directory, Terraform can be initialized as shown in Listing 6-12. The operation shown here downloads all the required plugins for the resources defined in the Terraform script.

$ terraform init

Initializing the backend...

Initializing provider plugins...

- Finding latest version of hashicorp/aws...

- Installing hashicorp/aws v3.54.0...

- Installed hashicorp/aws v3.54.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that Terraform can guarantee to make the same selections by default when you run "terraform init" in the future.

Terraform has been successfully initialized!

...........................

...........................

Listing 6-12

Initializing Terraform directory

You can optionally run terraform plan as shown in Listing 6-13 to see the actions Terraform will perform in your AWS account, that is, resources to be created, deleted, or modified.

$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:

  + create

Terraform will perform the following actions:

  # aws_codecommit_repository.test will be created

  + resource "aws_codecommit_repository" "test" {

      + arn             = (known after apply)

      + clone_url_http  = (known after apply)

      + clone_url_ssh   = (known after apply)

      + description     = "Example Description"

      + id              = (known after apply)

      + repository_id   = (known after apply)

      + repository_name = "devops-ios-repository"

      + tags_all        = (known after apply)

    }

Plan: 1 to add, 0 to change, 0 to destroy.

Listing 6-13

Terraform plan to see the actions Terraform will perform

After initializing the Terraform directory, and once you’re satisfied with the actions Terraform will take as shown in Listing 6-13, the resources can be created by applying the Terraform script as shown in Listing 6-14.

$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:

  + create

Terraform will perform the following actions:

  # aws_codecommit_repository.test will be created

  + resource "aws_codecommit_repository" "test" {

      + arn             = (known after apply)

      + clone_url_http  = (known after apply)

      + clone_url_ssh   = (known after apply)

      + description     = "Example Description"

      + id              = (known after apply)

      + repository_id   = (known after apply)

      + repository_name = "devops-ios-repository"

      + tags_all        = (known after apply)

    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?

  Terraform will perform the actions described above.

  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_codecommit_repository.test: Creating...

aws_codecommit_repository.test: Creation complete after 0s [id=devops-ios-repository]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Listing 6-14

Applying Terraform script to create CodeCommit repository

Creating with Python Boto3 SDK

AWS has SDK for different programming languages to enable you to manage and access AWS services with your preferred development language. Here I will focus on the AWS Python SDK called boto3. To learn more about boto3 SDK, visit the official documentation here: https://boto3.amazonaws.com/v1/documentation/api/latest/index.html.

Python is required on your workstation before boto3 can be installed. Assuming Python installed, Listing 6-15 shows how boto3 can be installed on a workstation using pip.

Refer to https://pip.pypa.io/en/stable/installation/ to install pip.

$  pip install boto3

Listing 6-15

Installing boto3

An example Python script that uses boto3 to create a CodeCommit repository is shown in Listing 6-16.

import boto3

client = boto3.client('codecommit')

response = client.create_repository(

    repositoryName='devops-ios-repository',

    repositoryDescription='Example Description'

)

print(response

Listing 6-16

Python script to create CodeCommit repository

Before boto3 can create resources, the workstation must be configured with AWS credentials. The credentials can be configured with AWS CLI, as shown in Listing 6-8.

Once the workstation is configured with AWS credentials, the Python script can then be executed, as shown in Listing 6-17.

Refer to www.python.org/downloads for information on downloading Python.

$ python codecommit.py

{

    u'repositoryMetadata': {

        u'repositoryName': u'devops-ios-repository',

        u'cloneUrlSsh': u'ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/devops-ios-repository',

        u'lastModifiedDate': datetime.datetime(2021, 8, 14, 21, 53, 50, 718000, tzinfo=tzlocal()),

        u'repositoryDescription': u'Example Description',

        u'cloneUrlHttp': u'https://git-codecommit.us-east-1.amazonaws.com/v1/repos/devops-ios-repository',

        u'creationDate': datetime.datetime(2021, 8, 14, 21, 53, 50, 718000, tzinfo=tzlocal()),

        u'repositoryId': u'3cd590f1-b898-43d7-b82c-eac6aef602a1',

        u'Arn': u'arn:aws:codecommit:us-east-1:123456789101:devops-ios-repository',

        u'accountId': u'123456789101'

    },

    'ResponseMetadata': {

        'RetryAttempts': 0,

        'HTTPStatusCode': 200,

        'RequestId': 'ff463517-219e-4d51-869f-fba5b1345c76',

        'HTTPHeaders': {

            'x-amzn-requestid': 'abcdefgh-219e-4d51-869f-fba5abcdefgh',

            'date': 'Sat, 14 Aug 2021 21:53:50 GMT',

            'content-length': '519',

            'content-type': 'application/x-amz-json-1.1'

        }

    }

}

Listing 6-17

Running Python script to create CodeCommit repository

Adding Source Code in AWS CodeCommit

As we know, CodeCommit is a private Git repository but as it is an AWS service, the AWS core fundamentals still apply, such as using identity access management (IAM) for authorization and authentication while connecting to the repository.

Let us explore the different options available to connect to a CodeCommit repository and store source code.

Connecting to a CodeCommit Repository

There are various options available to connect to a CodeCommit repository. The suitable option for you will depend on how you authenticate to use AWS services. For example, is it an IAM user, IAM Role, Federated access, etc.?

For each method to connect to a CodeCommit repository, the URL used is different; for a particular repository, the URL for each connection method can be retrieved as shown in Figure 6-6.

Figure 6-6

CodeCommit connection URLs

HTTPS

The HTTPS method is used with an IAM user, and it involves generating a Git username and password for that IAM user. The username and password generated are used as Git credentials when performing Git operations.

To generate a Git username and password for an IAM user, go to the Security credentials setting of the IAM user, as shown in Figure 6-7.

Figure 6-7

Security credentials settings of an IAM user

In the security credentials page for the IAM user, the option to generate HTTPS Git credentials for AWS CodeCommit is seen, as shown in Figure 6-8.

Figure 6-8

Generating HTTPS Git credentials

Once a credential is generated, you can download the credential as a .csv file or copy the username and password as shown in Figure 6-9.

Figure 6-9

Downloading generated HTTPS credential

The generated credential can be used to connect to a CodeCommit repository with its HTTPS URL by cloning a newly created repository, as shown in Listing 6-18.

$ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/devops-ios-repository

Cloning into 'devops-ios-repository'...

Username for 'https://git-codecommit.us-east-1.amazonaws.com': abdullahi-at-168181944806

Password for 'https://abdullahi-at-168181944806@git-codecommit.us-east-1.amazonaws.com':

warning: You appear to have cloned an empty repository.

$ cd devops-ios-repository/

$ git status

On branch master

No commits yet

nothing to commit (create/copy files and use "git add" to track)

Listing 6-18

Connect to CodeCommit repository with HTTPS

SSH

Similar to the HTTPS method, the SSH method of connecting to an AWS CodeCommit repository is also used for IAM users; this connection approach requires a private and public key pair. The public key is associated with the IAM user while the private is configured locally in the Git workstation. The first step to using the SSH connection method is to generate a SSH key pair using ssh-keygen, as shown in Listing 6-19.

$ ssh-keygen

Generating public/private rsa key pair.

Enter file in which to save the key (/home/cloudshell-user/.ssh/id_rsa): /home/cloudshell-user/.ssh/codecommit_key

Created directory '/home/cloudshell-user/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/cloudshell-user/.ssh/codecommit_key.

Your public key has been saved in /home/cloudshell-user/.ssh/codecommit_key.pub.

The key fingerprint is:

SHA256:AJ1EUBde/TT0K/D6SqaryVdSEalDPVN8K/Lp55yWPrU cloudshell-user@ip-10-1-34-183.ec2.internal

The key's randomart image is:

+---[RSA 2048]----+

|    o*+.o.oo=o.  |

|     .oo o *..oo |

|      . o ..+o..o|

|       . o oo....|

|        S o ooo. |

|         . ..o. .|

|          o+.   +|

|      . ..+ ...E.|

|       +oo....==.|

+----[SHA256]-----+

Listing 6-19

Generating SSH key pair

The key pair generated in Listing 6-19 is named codecommit_key, hence the public key generated will be named codecommit_key.pub, while the private key is codecommit_key, both stored in directory /home/cloudshell-user/.ssh.

Next you upload the public key to AWS for the intended IAM user. To upload a SSH public key for an IAM user, access the security credentials settings as shown in Figure 6-7. In the security credentials, the option to start the upload is as shown in Figure 6-10.

Figure 6-10

Start SSH public key upload

The content of the SSH public key will be provided as shown in Figure 6-11. In this example, this is the content of codecommit_key.pub. You can retrieve the content as shown in Listing 6-20.

$ cat ~/.ssh/codecommit_key.pub

Listing 6-20

Retriving SSH public key content

Figure 6-11

Adding SSH public key content

After successful public key upload, the SSH key ID is generated, as shown in Figure 6-12. Retrieve the SSH key ID as it will be used for local Git configuration.

Figure 6-12

SSH Key ID for the uploaded public key

The local Git configuration is stored in a config file in the .ssh folder on your workstation, as shown in Listing 6-21. The Host field in the config file is the CodeCommit URL pattern across all regions, the User field is the SSH user that should be used for connection, (this is the SSH key ID retrieved from AWS IAM as shown in Figure 6-12) and the IdentityFile field point to the location where the SSH private key is stored.

$ vim ~/.ssh/config

Host git-codecommit.*.amazonaws.com

User APKASOKDM2HTO6CTLRZS

IdentityFile ~/.ssh/codecommit_key

Listing 6-21

Local SSH configuration

Read and write permission should also be added to the config file permission as shown in Listing 6-22.

$ chmod 600 .ssh/config

Listing 6-22

Setting config file permission

Now with SSH configured, you can connect to the CodeCommit repository with its SSH URL by cloning a newly created repository, as shown in Listing 6-23.

$ git clone ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/devops-ios-repository

Cloning into 'devops-ios-repository'...

The authenticity of host 'git-codecommit.us-east-1.amazonaws.com (52.94.229.29)' can't be established.

RSA key fingerprint is SHA256:eLMY1j0DKA4uvDZcl/KgtIayZANwX6t8+8isPtotBoY.

RSA key fingerprint is MD5:a6:9c:7d:bc:35:f5:d4:5f:8b:ba:6f:c8:bc:d4:83:84.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'git-codecommit.us-east-1.amazonaws.com,52.94.229.29' (RSA) to the list of known hosts.

warning: You appear to have cloned an empty repository.

$ cd devops-ios-repository/

$ git status

On branch master

No commits yet

nothing to commit (create/copy files and use "git add" to track)

Listing 6-23

Connecting to CodeCommit repository with SSH

Git Remote CodeCommit (GRC)

This option is used when you use temporary credentials with IAM roles to connect with AWS. Common use cases for temporary credentials are identity federation, cross-account access, and roles for AWS services such as EC2.

GRC is a tool (git-remote-codecommit) that must be installed on a workstation, and in this approach, GRC takes the current AWS credentials of the user performing Git operations and uses them for authentication to CodeCommit; hence, no permanent IAM user is required.

GRC requires Python to be installed and can be installed with pip, as seen in Listing 6-24.

$ pip install git-remote-codecommit

Listing 6-24

Installing Git remote CodeCommit

To demonstrate how git-remote-codecommit works, I will assume an IAM role and use temporary credentials generated for the IAM role to connect to a repository.

For more information on creating and configuring IAM roles, review the AWS official documentation here: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html.

The IAM role that will be assumed is configured as in Figure 6-13.

Figure 6-13

IAM role with AWS CodeCommit permission

To get temporary credentials, I will assume the IAM role using the AWS CLI, as shown in Listing 6-25. Where the Role ARN, shown in Figure 6-13, is passed to the --role-arn flag and a descriptive name is provided for the session name in the --role-session-name flag.

$ aws sts assume-role --role-arn "arn:aws:iam::123456789101:role/CodeCommitRole" --role-session-name AWSCLI-Session

Listing 6-25

Assuming IAM role

This AWS CLI operation returns temporary credentials as output and the expiration information of the credentials. Retrieve the AccessKeyId, SecretAccessKey, SessionToken and configure the workstation where the Git operations will be performed, as shown in Listing 6-26.

$ export AWS_ACCESS_KEY_ID=<RoleAccessKeyID>

$ export AWS_SECRET_ACCESS_KEY=<RoleSecretAccessKey>

$ export AWS_SESSION_TOKEN=<RoleSessionToken>

Listing 6-26

Configure workstation with temporary credentials

Your workstation should now be using the temporary credentials. You can verify that the temporary credentials are configured correctly and are active, as shown in Listing 6-27.

$ aws sts get-caller-identity

{

    "UserId": "AROASOKDM2HTNVOJHAAV7:AWSCLI-Session",

    "Account": "123456789101",

    "Arn": "arn:aws:sts::123456789101:assumed-role/CodeCommitRole/AWSCLI-Session"

}

Listing 6-27

Verifying temporary credentials

After verifying that the temporary credential is active, I can connect to the repository using the GRC URL by cloning a newly created repository, as shown in Listing 6-28.

$ git clone codecommit::us-east-1://devops-ios-repository

Cloning into 'devops-ios-repository'...

warning: You appear to have cloned an empty repository.

$ cd devops-ios-repository/

$ git status

On branch master

No commits yet

nothing to commit (create/copy files and use "git add" to track)

Listing 6-28

Connecting to CodeCommit repository with Git remote CodeCommit

Pushing Code to CodeCommit Repository

Now that we have explored the different ways to connect to a CodeCommit repository by cloning an empty CodeCommit repository locally, let’s add some source code to the local repository and push the changes to CodeCommit.

The source code being used here is the sample app code, which can be downloaded from https://github.com/abdulola/devops-on-aws-ios-development-app.

After adding source code to the local directory but without committing to Git yet, the status of the repository can be checked as shown in Listing 6-29, where you will see the new files listed as untracked.

$ git status

On branch master

No commits yet

Untracked files:

  (use "git add <file>..." to include in what will be committed)

        SampleApp.xcodeproj/

        SampleApp/

        SampleAppTests/

        SampleAppUITests/

nothing added to commit but untracked files present (use "git add" to track)

Listing 6-29

Checking the status of new files added to local repository

The newly added files are then staged and committed to Git, as shown in Listing 6-30.

$ git add .

$ git commit -m "Add Sample Application Code"

[master (root-commit) 4c25bde] Add Sample Application Code

 34 files changed, 981 insertions(+)

 create mode 100644 SampleApp.xcodeproj/project.pbxproj

 create mode 100644 SampleApp.xcodeproj/project.xcworkspace/contents.xcworkspacedata

 create mode 100644 SampleApp.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist

 create mode 100644 SampleApp.xcodeproj/project.xcworkspace/xcuserdata/olaoyea.xcuserdatad/UserInterfaceState.xcuserstate

 create mode 100644 SampleApp.xcodeproj/xcuserdata/olaoyea.xcuserdatad/xcschemes/xcschememanagement.plist

 create mode 100644 SampleApp/Assets.xcassets/AccentColor.colorset/Contents.json

 create mode 100644 SampleApp/Assets.xcassets/AppIcon.appiconset/1024.png

 create mode 100644

...................................................

...................................................

...................................................

Listing 6-30

Committing source code to Git

The changes are then pushed to CodeCommit by simply running a Git push, as shown in Listing 6-31.

The CodeCommit SSH URL is used for this operation.

$ git push

Warning: Permanently added the RSA host key for IP address '52.94.233.146' to the list of known hosts.

Enumerating objects: 45, done.

Counting objects: 100% (45/45), done.

Delta compression using up to 2 threads

Compressing objects: 100% (40/40), done.

Writing objects: 100% (45/45), 75.21 KiB | 6.84 MiB/s, done.

Total 45 (delta 1), reused 0 (delta 0)

To ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/devops-ios-repository

 * [new branch]      master -> master

Listing 6-31

Pushing changes to CodeCommit

Migrating Git Repository to AWS CodeCommit

Existing remote Git repository can also be migrated to CodeCommit. To migrate to CodeCommit, the repository will be cloned locally and then pushed to the target CodeCommit repository.

As an example, I want to migrate a private GitHub repository named devops-on-aws-ios-development to a CodeCommit repository named target-codeommit-repository. First, I clone the source GitHub repository to a temporary local directory, which I named migrate-to-codecommit, as shown in Listing 6-32. The --mirror flag passed is used to clone a bare copy of the remote repository for migration.

$ git clone --mirror https://github.com/abdulola/devops-on-aws-ios-development.git migrate-to-codecommit

Cloning into bare repository 'migrate-to-codecommit'...

Username for 'https://github.com': abdulola

Password for 'https://abdulola@github.com':

remote: Enumerating objects: 97, done.

remote: Counting objects: 100% (97/97), done.

remote: Compressing objects: 100% (79/79), done.

remote: Total 97 (delta 11), reused 86 (delta 7), pack-reused 0

Unpacking objects: 100% (97/97), done.

Listing 6-32

Cloning source GitHub repository to local directory

Next, I will go into the local directory and then push the contents to the target CodeCommit repository, as shown in Listing 6-33. The --all flag ensures that all the branches for the repository are pushed to the destination CodeCommit repository.

$ cd migrate-to-codecommit/

$ git push --all ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/target-codeommit-repository

Enumerating objects: 97, done.

Counting objects: 100% (97/97), done.

Delta compression using up to 2 threads

Compressing objects: 100% (86/86), done.

Writing objects: 100% (97/97), 291.28 KiB | 10.40 MiB/s, done.

Total 97 (delta 11), reused 0 (delta 0)

To ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/target-codeommit-repository

 * [new branch]      main -> main

Listing 6-33

Pushing contents to the destination CodeCommit repository

At this point, the GitHub repository has been migrated to AWS CodeCommit, and the temporary local directory migrate-to-codecommit can be deleted. The newly migrated repository can be seen as shown in Figure 6-14.

Figure 6-14

Migrated repository in CodeCommit

To start interacting with the CodeCommit repository, you must clone the repository to local. Listing 6-34 shows the deleting of the temporary migration directory and cloning of the migrated repository locally.

The CodeCommit SSH URL is used for this operation.

$ rm -rf migrate-to-codecommit/

$ git clone ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/target-codeommit-repository

Cloning into 'target-codeommit-repository'...

Warning: Permanently added the RSA host key for IP address '52.94.226.180' to the list of known hosts.

remote: Counting objects: 97, done.

Receiving objects: 100% (97/97), 291.28 KiB | 15.33 MiB/s, done.

Resolving deltas: 100% (11/11), done .

Listing 6-34

Cloning migrated repository to local

Branching in AWS CodeCommit

There are different ways to create a branch in CodeCommit, when you make your first commit to a CodeCommit repository, a default branch is created named main. Next, I will show you how to create additional branches using the AWS Console, Git, and AWS CLI.

Creating Branch on AWS Console

From within your CodeCommit repository console, select the Branches menu as shown in Figure 6-15.

The repository migrated to CodeCommit in the previous section is used for this exercise.

Figure 6-15

Accessing the Branches menu from the CodeCommit repository console

Start branch creation on the next page as shown in Figure 6-16. As mentioned earlier, the default main branch for the repository already exists.

Figure 6-16

Start branch creation

When creating a branch, you must select the branch you want to create the new branch from. As shown in Figure 6-17, a branch named dev is being created from the only existing branch main.

Figure 6-17

Creating new dev branch

The new dev branch created can be seen in Figure 6-18. The last commit message for both branches is the same, and this is because the dev branch is freshly created, and no additional commits have been added to it.

Figure 6-18

New dev branch created

Creating Branch with Git

New branches in CodeCommit can also be created using Git from the Git command line. When a branch is created from the Git command line, initially it is only created locally, the branch must be pushed to CodeCommit for it to be created on CodeCommit.

Listing 6-35 shows the creation of a new branch locally and a push operation to create this branch on CodeCommit.

The repository migrated to CodeCommit in the previous section is used for this exercise. The local repository is already linked to the CodeCommit repository from the Git clone operation in Listing 6-34.

$ cd target-codeommit-repository/

$ git checkout -b test

Switched to a new branch 'test'

$ git push origin test

Total 0 (delta 0), reused 0 (delta 0)

To ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/target-codeommit-repository

 * [new branch]      test -> test

Listing 6-35

Create CodeCommit branch using Git

The newly created branch test can be seen on the CodeCommit console, as shown in Figure 6-19.

Figure 6-19

New test branch created from Git

Creating Branch with AWS CLI

To create a branch using the AWS CLI, in addition to passing the repository name and branch name as arguments, the Git commit ID you want the branch to be created from must also be specified. To retrieve a Git commit ID, you can access all commits for your CodeCommit repository, as shown in Figure 6-20.

Figure 6-20

Access all commits for CodeCommit repository

You can copy the commit ID to create the new branch from, as shown in Figure 6-21.

Figure 6-21

Copying commit ID

With the commit ID to use for branch creation known, a new branch can be created using AWS CLI, as shown in Listing 6-36.

$ aws codecommit create-branch --repository-name target-codeommit-repository --branch-name preprod --commit-id 7da70dd7ef5eb237012ac0699367922d9cb3524e

Listing 6-36

Creating branch using AWS CLI

The new preprod branch created can be seen on the CodeCommit console as shown in Figure 6-22.

Figure 6-22

New preprod branch created using AWS CLI

Updating Local Repository

With all the new branches created, some branches exist on CodeCommit but not locally, you can fetch updates for the remote repository and view all branches as shown in Listing 6-37.

$ git remote update origin

Fetching origin

From ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/target-codeommit-repository

 * [new branch]      preprod    -> origin/preprod

$ git branch --all

  dev

* test

  remotes/origin/HEAD -> origin/dev

  remotes/origin/dev

  remotes/origin/main

  remotes/origin/preprod

  remotes/origin/test

Listing 6-37

Fetch update from remote repository

Configuring Default Branch

When you make your first commit to a CodeCommit repository, a default base branch named main is created for you. This default branch can be changed if desired to other branches in the repository. To change a default branch, go to the repository settings as shown in Figure 6-23.

Figure 6-23

Repository settings

In the Default branch menu, the new default can be selected from the list of repository branches, as shown in Figure 6-24.

Figure 6-24

Select default branch

Pull Requests in AWS CodeCommit

AWS CodeCommit repositories support creating pull requests before merging branches. There are also some additional pull request features specific to CodeCommit such as Approval rules and Amazon CodeGuru Reviewer integration. Let’s explore all these concepts next.

Creating a Pull Request

When two branches are to be merged, for example, a feature branch merging with a main branch, a pull request can either be created on the AWS console or the CLI.

Before you can create a pull request to merge two branches, there must be a delta between the two branches. A delta is introduced into a test branch in the local repository and pushed to the remote codecommit repository, as shown in Listing 6-38.

$ echo "Add New Text" >> README.md

$ git status

On branch test

Changes not staged for commit:

  (use "git add <file>..." to update what will be committed)

  (use "git restore <file>..." to discard changes in working directory)

        modified:   README.md

no changes added to commit (use "git add" and/or "git commit -a")

$ git add README.md

$ git commit -m "Add new text to Readme"

[test 7ea6176] Add new text to Readme

 1 file changed, 1 insertion(+)

$ git push --set-upstream origin test

Enumerating objects: 5, done.

Counting objects: 100% (5/5), done.

Delta compression using up to 2 threads

Compressing objects: 100% (2/2), done.

Writing objects: 100% (3/3), 311 bytes | 311.00 KiB/s, done.

Total 3 (delta 1), reused 0 (delta 0)

To ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/target-codeommit-repository

   7da70dd..7ea6176  test -> test

Branch 'test' set up to track remote branch 'test' from 'origin'.

Listing 6-38

Adding changes to repository test branch and pushing to CodeCommit

Now that code changes have been pushed to the test branch of the CodeCommit repository, a pull request can be created.

AWS Console

To create a pull request, you can access the pull request page for your repository, as shown in Figure 6-25.

Figure 6-25

Access pull request console

Once on the Pull requests page of a repository, you can start the pull request creation as shown in Figure 6-26.

Figure 6-26

Starting pull request creation

You select the Source and Destination branch as shown in Figure 6-27. For this example, the source branch that contains the code change is test and the destination branch is main.

Figure 6-27

Selecting Source and Destination branch

Provide a title and optional description for the pull request, as shown in Figure 6-28. Typically, the description will contain details of the code changes to provide context for team members that will be reviewing it.

Figure 6-28

Providing title and description for pull request

You can review the changes introduced by the source branch before finally creating the pull request, as shown in Figure 6-29.

Figure 6-29

Reviewing changes introduced by the pull request

And then, finally, once the pull request is created successfully, the new pull request will look like in Figure 6-30.

Figure 6-30

Pull request created

AWS CLI

To create a pull request on the AWS CLI, we use the command in Listing 6-39. The pull request title is passed to the --title option, the description to the --description option, a unique idempotency token which you generated should be passed to the --client-request-token option, and the repository and branch information is passed to the --targets option.

$ aws codecommit create-pull-request --title "Merge Test to Main Branch" --description "Example description" --client-request-token Example --targets repositoryName=target-codeommit-repository,sourceReference=Test,destinationReference=main

Listing 6-39

Creating pull request with the AWS CLI

Merging Pull Request

Once the pull request has been reviewed by others and ready to be merged, this operation can also be performed on the AWS Console or the CLI.

AWS Console

As far as the two branches are mergeable, that is, there are no merge conflicts or approval rules that will prevent the merge, the Merge option is always activated and can be seen as shown in Figure 6-31.

Figure 6-31

Starting pull request merge

To merge, you must select a Merge strategy; CodeCommit supports three different merge strategies, as shown in Figure 6-32. The default Git behavior for merging is fast forward when possible. If a particular merge type cannot be applied due to the merge scenario, this option will be grayed out and cannot be selected. You can optionally choose to delete the source branch after a successful merge, with this option selected, the test branch used in this example will be deleted in the repository.

Figure 6-32

Selecting merge type and merge pull request

Once the pull request has been merged successfully, you get a confirmation, as shown in Figure 6-33.

Figure 6-33

Pull request merged

To see the merged changes that were applied, you can switch to the destination branch in your repository and inspect the file(s) changes made by the pull request. As shown in Figure 6-34, the README.md of the repository, which was updated by the pull request, has been changed.

Figure 6-34

Updated file from the pull request

Now that the remote main branch has been updated, to keep the local main branch up to date as well, a git pull operation must be performed, as shown in Listing 6-40.

$ git pull

From ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/target-codeommit-repository

   7da70dd..7ea6176  main       -> origin/main

Your configuration specifies to merge with the ref 'refs/heads/test'

from the remote, but no such ref was fetched.

$ git checkout main

Branch 'main' set up to track remote branch 'main' from 'origin'.

Switched to a new branch 'main'

Listing 6-40

Updating local main branch with changes from remote

AWS CLI

To merge a pull request on the AWS CLI, the command to execute depends on the merge strategy you want to apply. Listing 6-41 shows how to merge using the fast forward merge strategy. The pull request ID is passed to the --pull-request-id option, the full commit ID which is the latest from the source branch is passed to the --source-commit-id option, and, lastly, the repository name is passed to the --repository-name option.

$ aws codecommit merge-pull-request-by-fast-forward --pull-request-id 1 --source-commit-id EXAMPLE1112222333 --repository-name target-codeommit-repository

Listing 6-41

Merging using fast forward strategy

To see how to use the CLI with other merge strategies, see https://docs.aws.amazon.com/cli/latest/reference/codecommit/.

Creating Approval Rule Template

Approval rules template allows you to automatically apply approval rules to pull requests created in repositories. In each AWS region, you can apply an approval rule template to some, or all, pull requests created in all CodeCommit repositories.

With approval rules templates, you can automatically apply an approval rule when a pull request is created for a given branch, for example, your main branch, and require approval from certain team members before the pull request can be merged.

Let’s explore how an approval template can be created on the AWS Console and the CLI.

AWS Console

You can access the approval rule template page on the AWS console, as shown in Figure 6-35.

Figure 6-35

Accessing Approval rule templates page

And you can start creating an approval rule template, as shown in Figure 6-36.

Figure 6-36

Start creating an approval rule template

The approval rule template I am creating is defined as shown in Figure 6-37. The minimum number of approvals is defined as 1, the list of valid approvers is also provided. Here I am providing the ARN of an IAM user; the destination branch to apply a rule to is also specified. Finally, the list of repositories to apply this template to is specified.

Figure 6-37

Approval rule template definition

AWS CLI

To create an approval rule template on the AWS CLI, the template must first be defined in JSON, as shown in Listing 6-42. The example template defined is applied to the main branch of repositories, requires two approvals, and an IAM user ARN is provided in the list of approvers.

{

  "Version": "2018-11-08",

  "DestinationReferences": [

    "refs/heads/main"

  ],

  "Statements": [

    {

      "Type": "Approvers",

      "NumberOfApprovalsNeeded": 2,

      "ApprovalPoolMembers": [

        "arn:aws:iam::123456789101:user/abdullahi"

      ]

    }

  ]

}

Listing 6-42

Approval rule template definition

The template can be provided to the AWS CLI command as shown in Listing 6-43. The approval template name is provided in the --approval-rule-template-name option, a description is provided in the --approval-rule-template-description option, and the template JSON definition is provided in the --approval-rule-template-content option.

$ aws codecommit create-approval-rule-template --approval-rule-template-name Engineering-Manager-Approval --approval-rule-template-description "Requires Engineering Manager Approval" --approval-rule-template-content "{\"Version\":\"2018-11-08\",\"DestinationReferences\":[\"refs\/heads\/main\"],\"Statements\":[{\"Type\":\"Approvers\",\"NumberOfApprovalsNeeded\":2,\"ApprovalPoolMembers\":[\"arn:aws:iam::123456789101:user\/abdullahi\"]}]}"

{

    "approvalRuleTemplate": {

        "approvalRuleTemplateId": "d812838a-4010-4c68-80ce-e03b8ba456c7",

        "approvalRuleTemplateName": "Engineering-Manager-Approval",

        "approvalRuleTemplateDescription": "Requires Engineering Manager Approval",

        "approvalRuleTemplateContent": "{\"Version\":\"2018-11-08\",\"DestinationReferences\":[\"refs/heads/main\"],\"Statements\":[{\"Type\":\"Approvers\",\"NumberOfApprovalsNeeded\":2,\"ApprovalPoolMembers\":[\"arn:aws:iam::123456789101:user/abdullahi\"]}]}",

        "ruleContentSha256": "609e9411c4a0a04d8f49438ac4faa377393355ba831646637f5895a4283aa732",

        "lastModifiedDate": "2021-08-18T00:23:22.780000+00:00",

        "creationDate": "2021-08-18T00:23:22.780000+00:00",

        "lastModifiedUser": "arn:aws:iam::123456789101:user/abdullahi"

    }

}

Listing 6-43

Creating an approval rule template from the AWS CLI

Pull Request with Approval Rule Template

If you create a pull request and an approval rule template has been applied to the repository and the branch defined in the approval rule template matches the destination branch of pull request, without any further action, an approval rule is created for the pull request. The condition of the approval rule is as defined in the approval rule template.

Figure 6-38 shows a pull request created to merge the dev branch to main branch of our example CodeCommit repository. An approval rule is applied to this pull request because the approval rule template created as shown in Figure 6-37 applies to this repository and the main branch. Because I am logged in to the AWS Console with an IAM user specified in the list of approvers in the approval rule template, I see the Approve option available.

Figure 6-38

Pulling request with approval rule applied

Once the pull request has been approved, the approval rules requirement has been fulfilled and the Merge button appears as shown in Figure 6-39.

Figure 6-39

Merging pull request after approval

Amazon CodeGuru Reviewer Integration

Amazon CodeGuru reviewer is an AWS service that detects defects in code using machine learning, CodeGuru reviewer can be associated with a CodeCommmit repository. When a pull request is created in a repository associated with a CodeGuru reviewer, it automatically starts reviewing the pull request and adds findings and recommendations as comments in the pull request.

Currently CodeGuru reviewer only supports reviewing Java and Python code.

Associating CodeGuru Reviewer

To associate CodeGuru reviewer with a CodeCommit repository, access the Amazon CodeGuru Reviewer tab in the repository settings, as shown in Figure 6-40.

Figure 6-40

CodeGuru Reviewer tab in CodeCommit repository settings

Associate CodeGuru Reviewer to your CodeCommit repository, as shown in Figure 6-41.

Figure 6-41

Associating CodeGuru reviewer

The association doesn’t complete immediately, while associating the status is Associating. As shown in Figure 6-42, the association can take up to 10 minutes.

Figure 6-42

CodeGuru Reviewer association in progress

Once the association is complete, the status changes to Associated. The option to disassociate the repository also appears, as shown in Figure 6-43.

Figure 6-43

CodeGuru Reviewer associated

Refer to https://docs.aws.amazon.com/codeguru/latest/reviewer-ug/create-codecommit-association.html#create-codecommit-association-cli on how to associate Amazon CodeGuru Reviewer to a repository via the AWS CLI.

Reviewing a Pull Request

Let’s see CodeGuru Reviewer in action by creating a pull request with a Python code. To trigger a CodeGuru reviewer comment, I will use a Python code with a defect. The Python code to be used is shown in Listing 6-44; this is a simple code to list all the objects in an S3 bucket.

import boto3

s3_client = boto3.client('s3')

s3_client.list_objects(Bucket="BucketName")

Listing 6-44

Example Python code to trigger CodeGuru reviewer

I will add this code snippet to a .py file in my repository’s dev branch and create a pull request to merge to the main branch. Automatically, CodeGuru reviewer starts reviewing the pull request, this can be seen as shown in Figure 6-44.

Figure 6-44

CodeGuru Reviewer reviewing pull request

After completing its review, it leaves a comment if any defect is found in the pull request. As seen in Figure 6-45, I am using an outdated API in my code and CodeGuru reviewer recommends using the revised API.

Figure 6-45

Reviewing CodeGuru comments

Summary

The automating and orchestrating of iOS application development processes starts with source control management. In this chapter, we covered the fundamentals of Git version control system and then dove deep into the Git features of AWS CodeCommit and how it can be used as a private remote Git repository, with walkthrough examples. We then covered its integration with Amazon CodeGuru Reviewer for automatically reviewing Python and Java code with machine learning.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

A. OlaoyeBeginning DevOps on AWS for iOS Developmenthttps://doi.org/10.1007/978-1-4842-8023-2_7

7. Integrating AWS CodeCommit with Jenkins

Abdullahi Olaoye1  

(1)

McKinney, TX, USA

When application source code is stored in AWS CodeCommit for source control, it can be integrated with other build and test systems for continuous integration such that when new commits are made in the source repository, automated build and/or test is triggered.

In this chapter, we will see how AWS CodeCommit can be integrated with Jenkins using Jenkins plugins. With this integration, iOS application build jobs can be set up on Jenkins using AWS CodeCommit for source control where build is triggered on Jenkins when code is committed to AWS CodeCommit.

All example code used in this chapter can be found at https://github.com/abdulola/devops-on-aws-ios-development/tree/main/Chapter_7.

Jenkins CodeCommit Plugin

Plugins are used to enhance the functionality of Jenkins to suit specific use cases, it allows the integration of various tools and Cloud providers with Jenkins. To integrate AWS CodeCommit and Jenkins, an AWS CodeCommit plugin is used.

The Jenkins AWS CodeCommit plugin works by sending messages to an Amazon Simple Queue Service (SQS) queue when a commit is made through Amazon Simple Notification Service (SNS). The Jenkins job periodically polls the queue for any new messages and if found it triggers a new build for the job.

To install this plugin on a Jenkins controller, Figure 7-1 shows how to access the Manage Jenkins page from a Jenkins environment home page.

Figure 7-1

Accessing the Manage Jenkins console

On the settings console, there are various options for configuring a Jenkins environment, as shown in Figure 7-2; you can access the Manage Plugins page to manage plugins.

Figure 7-2

Managing Jenkins plugins

All plugins installed in a Jenkins environment can be managed on this page as seen from the different available tabs. The Updates tab shows any plugins installed that are due for updates, the Available tab shows all available plugins that can be installed in a Jenkins environment, the Installed tab shows all the Jenkins plugins installed in that Jenkins environment, and the Advanced tab provides options to perform actions like installing plugins by manually uploading an archive. A use case for using the Advanced option is when you want to install the older version of a plugin not available via the web UI.

To start installing the AWS CodeCommit plugin, switch to the Available tab and search for CodeCommit, as shown in Figure 7-3.

Figure 7-3

Searching for AWS CodeCommit plugin

Select AWS CodeCommit Trigger and install, as shown in Figure 7-4. You can instruct Jenkins to either install instantly without restarting or download and wait for next restart to install the plugin.

Figure 7-4

Installing AWS CodeCommit plugin

Now that the plugin is installed, to start using the plugin in projects, it must be configured. However, to configure the plugin, some AWS resources are required to be set up as a prerequisite.

In the next section, we will see how these prerequisite resources can be set up.

Setting Up Integration Components

To configure the AWS CodeCommit Jenkins plugin installed in the previous section, we require an Amazon SQS queue and Amazon SNS topic to be created and the CodeCommit repository configured to publish messages to the SQS queue when new commits are created.

Setting Up Components via AWS Console

In this section, we will set these components up manually via the AWS Console.

Creating SQS Queue

To access the Amazon Simple Queue Service (SQS) console, search for SQS on the AWS console, as shown in Figure 7-5.

Figure 7-5

Searching for SQS on the AWS Management Console

Start SQS queue creation as shown in Figure 7-6.

Figure 7-6

Starting SQS queue creation

Figure 7-7 shows an example of how to configure the SQS queue. A standard queue type is sufficient for this use case.

Figure 7-7

Configuring and creating an SQS queue

Creating SNS Topic and Subscribing SQS Queue

To access the Amazon Simple Notification Service (SNS) console, search for SNS on the AWS console, as shown in Figure 7-8.

Figure 7-8

Search for SNS on AWS Management Console

Start creating an SNS topic as shown in Figure 7-9.

Figure 7-9

Starting the SNS topic creation

Example configuration for the SNS topic can be seen in Figure 7-10. A standard SNS topic type is suitable for this use case.

Figure 7-10

Configuring and createing an SNS topic

Now that an SQS queue and an SNS topic have been created, the two will be integrated together by subscribing the SQS queue to the SNS topic. The SNS topic publishes messages to all its subscribed entities; when the SQS queue receives a message from the SNS topic, it stores the message until a consumer consumes the message from the queue. The consumer in this case is the AWS CodeCommit plugin running in Jenkins.

Access the SNS topic created and create a subscription as shown in Figure 7-11.

Figure 7-11

Starting subscription creation

Since the subscription is created for an SQS queue, the Amazon SQS protocol and the respective SQS queue should be selected accordingly, as shown in Figure 7-12.

Figure 7-12

Configuring and creating an Amazon SQS subscription

An example of a successfully created SQS subscription is shown in Figure 7-13.

Figure 7-13

Successfullly created SQS subscription

Setting Up SQS Queue Permission

To allow SNS topics to publish messages to an SQS queue, access policy on the SQS queue should be configured to allow access to the SNS topic. To configure the SQS queue access policy, access the Access policy tab from the SQS queue, as shown in Figure 7-14.

Figure 7-14

Configuring SQS queue access policy

By default, the IAM identity in an account has access to the SQS queue created, but to allow access to AWS services such as SNS, the access policy would be modified.

Listing 7-1 shows an example policy that can be applied for access policy to allow an SNS topic to publish messages to an SQS queue.

{

  "Version": "2008-10-17",

  "Id": "__default_policy_ID",

  "Statement": [

    {

      "Sid": "__owner_statement",

      "Effect": "Allow",

      "Principal": {

        "AWS": "arn:aws:iam::123456789101:root"

      },

      "Action": "SQS:*",

      "Resource": "arn:aws:sqs:us-east-1:123456789101:codecommit-trigger-queue"

    }

  ,

  {

    "Effect":"Allow",

    "Principal": {

      "Service": "sns.amazonaws.com"

    },

    "Action":"sqs:SendMessage",

    "Resource":"arn:aws:sqs:us-east-1:123456789101:codecommit-trigger-queue",

    "Condition":{

      "ArnEquals":{

        "aws:SourceArn":"arn:aws:sns:us-east-1:123456789101:codecommit-trigger-topic"

      }

    }

  }]

}

Listing 7-1

SQS queue access policy

To use this example policy, change Account IDs and resource names to match yours.

Testing SQS and SNS Integration

With the SQS queue access policy configured, the integration of Amazon SQS and Amazon SNS is complete. The integration can be tested by manually publishing a message in the SNS topic and verifying that the message is received on the SQS queue.

Publish message on SNS topic as shown in Figure 7-15.

Figure 7-15

SNS publish message

Provide a test message subject and message body as shown in Figure 7-16. This message will be fanned out to all entities subscribed to the SNS topic.

Figure 7-16

Providing test message subject and body

Switch to the SQS queue console to verify that the message is received. On the SQS queue, you can start receiving the message as shown in Figure 7-17.

Figure 7-17

Receiving message on SQS queue

Messages can be polled for and viewed as shown in Figure 7-18. All the messages present on the queue will be shown. To poll for messages, click on the Poll for messages option, as shown in Figure 7-18.

Figure 7-18

Polling and viewing messages on SQS queue

The message on the queue verifies that the message published from SNS is delivered to SQS.

To view a message content, click on a message ID as shown in Figure 7-18. The message content would look like Listing 7-2.

{

  "Type": "Notification",

  "MessageId": "f1ddce6c-0f19-5961-8413-97ad8d705b83",

  "TopicArn": "arn:aws:sns:us-east-1:123456789101:codecommit-trigger-topic",

  "Subject": "Test Message",

  "Message": "Test Message",

  "Timestamp": "2021-09-19T18:23:37.676Z",

  "SignatureVersion": "1",

  "Signature": "PPp9ZRsxc8/TY**************",

  "SigningCertURL": "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-7ff5318490ec183fbaddaa2a969abfda.pem",

  "UnsubscribeURL": "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:123456789101:codecommit-trigger-topic:7272ea84-728e-4fdb-9e15-6a02e9c5b2f1"

}

Listing 7-2

SQS message received from SNS topic

Configuring CodeCommit Repository Trigger

As shown in the previous sections, an SQS queue and an SNS topic have been created. The last component to set up is configuring triggers on AWS CodeCommit. CodeCommit trigger allows you to trigger actions when repository events happen.

You can configure specific repository events or all repository events to trigger an action. For actions, you can invoke an AWS lambda function or send notification to an Amazon SNS topic. We will be looking at how repository events can send notification to an Amazon SNS topic.

To start, from the CodeCommit repository you wish to use, go to the Settings page, as shown in Figure 7-19.

Figure 7-19

CodeCommit repository settings

Create a trigger from the Triggers tab, as shown in Figure 7-20.

Figure 7-20

Start CodeCommit trigger creation

Configure and create the trigger as shown in Figure 7-21. For SNS topic, select the SNS topic that has been integrated to an SQS queue for the same purpose.

Figure 7-21

Configuring and creating CodeCommit trigger

Setting Up Components via Terraform

Let’s set up the required AWS resources via Terraform for Infrastructure as Code (IaC).

The snippet of an example Terraform code to set up all the required resources is shown in Listing 7-3.

The full code can be retrieved from https://github.com/abdulola/devops-on-aws-ios-development/blob/main/Chapter_7/Listing%207-3.tf.

variable "aws-account-id" {

  default = "xxxxxxxxxxxx"

}

.

.

.

provider "aws" {

  region = "${var.aws-region}"

  alias = "default"

}

data "aws_iam_policy_document" "sns-sqs-policy" {

  policy_id = "arn:aws:sqs:us-east-1:${var.aws-account-id}:testing/SQSDefaultPolicy"

  statement {

    sid = "SubscribeToSNS"

    effect = "Allow"

    principals {

      type = "AWS"

      identifiers = [ "*" ]

    }

    actions = [ "SQS:SendMessage" ]

    resources = [ "${aws_sqs_queue.main.arn}" ]

    condition {

      test = "ArnLike"

      variable = "aws:SourceArn"

      values = [ "arn:aws:sns:${var.aws-region}:${var.aws-account-id}:${var.sns-topic-prefix}*${var.sns-topic-suffix}" ]

    }

  }

}

.

.

.

resource "aws_sqs_queue" "main" {

  name = "codecommit-notifications-queue"

  delay_seconds = 90

  max_message_size = 2048

  message_retention_seconds = 86400

  receive_wait_time_seconds = 10

}

resource "aws_sqs_queue_policy" "sns" {

  queue_url = "${aws_sqs_queue.main.id}"

  policy = "${data.aws_iam_policy_document.sns-sqs-policy.json}"

}

.

.

.

output "sns-name" {

  value = "${aws_sns_topic.main.name}"

}

output "sns-arn" {

  value = "${aws_sns_topic.main.arn}"

}

Listing 7-3

Setting up components with Terraform

To create the resources, save this Terraform code into a file with .tf extension and initialize, as shown in Listing 7-4 .

Replace the variable repository_name in Listing 7-3 with your CodeCommit repository name.

$ terraform init

Initializing the backend...

Initializing provider plugins...

- Finding latest version of hashicorp/aws...

- Installing hashicorp/aws v3.59.0...

- Installed hashicorp/aws v3.59.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that Terraform can guarantee to make the same selections by default when you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work.

If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.

Listing 7-4

Initializing Terraform

After successful initialization, all dependencies have been downloaded, now you can run terraform apply to deploy the resources. The output of the command would look like the example snippet shown in Listing 7-5.

$ terraform apply

provider.aws.region

  The region where AWS operations will take place. Examples

  are us-east-1, us-west-2, etc.

  Enter a value: us-east-1

.

.

.

Do you want to perform these actions?

  Terraform will perform the actions described above.

  Only 'yes' will be accepted to approve.

  Enter a value: yes

.

.

.

Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

Outputs:

sns-arn = "arn:aws:sns:us-east-1:123456789101:codecommit-target-codeommit-repository-topic"

sns-name = "codecommit-target-codeommit-repository-topic"

Listing 7-5

Creating resources with Terraform

Configuring Plugin

The plugin prerequisite resources that have been set up are SQS queue, SNS topic, subscribing SQS queue to SNS topic, and setting up CodeCommit trigger to send notification to the plugin SNS topic on repository events. Now the plugin can be configured to utilize the resources that have been created to complete the integration.

To configure the plugin, access the Jenkins system configuration as shown in Figure 7-22.

Figure 7-22

Configuring Jenkins system

On the system configuration page, the CodeCommit plugin configuration would be visible. Figure 7-23 shows how you can start configuring the plugin with the already created SQS queue.

Figure 7-23

Adding SQS config to CodeCommit plugin

The AWS region and SQS queue name should be provided as shown in Figure 7-24. The plugin tests access to the SQS queue using the AWS credentials provided.

Figure 7-24

Configuring SQS queue name and AWS region

If available, existing AWS credentials configured on Jenkins can be used, but to add new AWS credentials, an example is shown in Figure 7-25.

Figure 7-25

Adding AWS credentials to Jenkins

The AWS credentials configured for the plugin must have SQS access and CodeCommit Read only access.

The Jenkins CodeCommit Plugin should now be configured and active. Next, we will create a sample Jenkins job to verify how the plugin works.

Creating Jenkins Job with AWS CodeCommit Source

In this section, we will see how an example Jenkins job can use AWS CodeCommit source and how to use the configured Jenkins CodeCommit plugin for automated builds. Figure 7-26 shows how to start the creation of a Jenkins job.

Figure 7-26

Starting creation of a Jenkins job

We will create a Jenkins Freestyle project. As shown in Figure 7-27, a project name should be provided.

Figure 7-27

Creating a Jenkins freestyle project

For project configuration, the CodeCommit repository configured earlier in this chapter should be added to the Source Code Management (SCM), as shown in Figure 7-28.

Figure 7-28

Configuring SCM with AWS CodeCommit

If available, existing CodeCommit Git credentials configured on Jenkins can be used, but to add new Git credentials, an example is shown in Figure 7-29. The Git credentials provided here is the HTTPS credentials for the CodeCommit IAM user.

Figure 7-29

Configure CodeCommit HTTPS credentials

Configure the build trigger for automated builds as shown in Figure 7-30. The SQS queue that will be monitored by the plugin is shown.

Figure 7-30

Setting up build trigger

For the build, some basic example commands are entered, as shown in Figure 7-31, and the project is created. When a build is triggered, the contents of the repository are cloned to the Jenkins workspace, the commands provided in Figure 7-31 list the workspace and confirm if the repository contents are downloaded.

Figure 7-31

Entering build commands

An example freestyle project can be seen in Figure 7-32.

Figure 7-32

Jenkins freestyle project

Testing AWS CodeCommit Plugin

To test the plugin, a new commit will be created in the CodeCommit repository, which will then trigger automated build on Jenkins.

Listing 7-6 shows the creation of a new commit in the CodeCommit repository configured as the source of the example Jenkins job.

$ touch testfile

$ git add . && git commit -m "Test listing directory"

[master fb68e27] Test listing directory

 1 file changed, 0 insertions(+), 0 deletions(-)

 create mode 100644 testfile

$ git push

Enumerating objects: 4, done.

Counting objects: 100% (4/4), done.

Delta compression using up to 6 threads

Compressing objects: 100% (2/2), done.

Writing objects: 100% (3/3), 285 bytes | 285.00 KiB/s, done.

Total 3 (delta 1), reused 0 (delta 0), pack-reused 0

To codecommit::us-east-1://devops-ios-repository

   bc42da1..fb68e27  master -> master

Listing 7-6

Adding commit to AWS CodeCommit Repository

On the Jenkins project, you can verify that the project was automatically started from that new commit as shown in Figure 7-33.

Figure 7-33

Jenkins job automated build

Figure 7-34 shows the build log which contains the output of all the build commands.

Figure 7-34

Jenkins job build output

Summary

Jenkins is a very widely used tool within DevOps due to its rich features and ability to integrate with a lot of other tools to satisfy different use cases, which is why it is also useful for iOS development. When source code for iOS application is stored on AWS CodeCommit, it becomes crucial to integrate CodeCommit with Jenkins to enable end-to-end automation.

So far, we have dived deep into the process of integrating Jenkins and CodeCommit for automated Jenkins build with a basic build. Next, we will pick up from here and add more iOS-specific builds and testing to Jenkins.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

A. OlaoyeBeginning DevOps on AWS for iOS Developmenthttps://doi.org/10.1007/978-1-4842-8023-2_8

8. Automated Build, Test, and Release with Fastlane

Abdullahi Olaoye1  

(1)

McKinney, TX, USA

In the previous chapter, we learnt how application source code stored in AWS CodeCommit can be integrated with Jenkins jobs to enable automated builds when code changes are made. Having a build system that is automatically invoked when the code changes is a key requirement for Continuous Integration and Continuous Delivery or Deployment (CICD), so in this chapter, we will take advantage of automated builds and dive deep into how you can use Jenkins and Fastlane to automate build, test, and release of iOS applications.

All example code used in this chapter can be found at https://github.com/abdulola/devops-on-aws-ios-development/tree/main/Chapter_8.

Setting Up Code Signing with Fastlane match and Amazon S3

Code signing in iOS development is a mechanism of signing iOS applications before they can be distributed or uploaded to app store. It is required for preserving integrity as it ensures that an iOS app is from a known, trusted source and has not been modified since it was last signed. Ideally, each developer will have a signing identity, but this creates problems and adds administrative overhead when a team comprises of multiple developers, and they must manage multiple signing identity.

With Fastlane match, a development team can share one code signing identity to help simply code signing. It creates all the required signing resources such as certificates, provisioning profiles and stores them in a centralized location where every team member can then retrieve these credentials and use for code signing. When using Fastlane for app development and release, this becomes seamless because the process of retrieving the credentials from the centralized location and configuring the build system is inbuilt in Fastlane. This is particularly useful when using Fastlane in a Continuous Integration and Continuous Delivery or Deployment (CICD) system because the credentials can be retrieved from the centralized location automatically and applications built seamlessly.

For centralized storage for code signing credentials, let’s see how Amazon S3 can be used as a storage location for Fastlane match. Later in the chapter, we will see how these credentials are retrieved from S3 during application build for signing build artifacts.

For more detailed information on Fastlane codesigning concepts see https://codesigning.guide/.

Initializing Fastlane match

Fastlane match must first be set up on a local development workstation within the project folder. But first Fastlane must be installed on the development workstation. There are various ways of installing Fastlane, but the easiest method is using Homebrew, as seen in Listing 8-1.

$ brew install fastlane

Listing 8-1

Installing Fastlane with Homebrew

After Fastlane installation, initialize match from within the iOS project directory, as shown in Listing 8-2.

For the storage mode, as seen in Listing 8-2, match supports different storage modes, but we will focus on using S3 as a storage mode so select S3.

Listing 8-2. Initializing match

If you get an “Unable to locate Xcode” error from the command, open Xcode Preferences and in Location ➤ Command Line Tools, select the Xcode CLI (command line interface) version and run the Fastlane command again.

When initialized, it creates a file named Matchfile in the project directory. This file will contain all the necessary information needed to successfully execute match. Initially this file won’t have all the information, but we will configure the required information next.

We first create an S3 bucket for the centralized storage and configure AWS credentials environment variables that will allow Fastlane to access the S3 bucket, as shown in Listing 8-3. A unique match password is also configured; this password is used to encrypt/decrypt certificates.

The match password is any passphrase chosen by the user and used as an additional layer of security.

Note down the bucket name, the environment variables name used for the AWS credentials, and the match password.

$ aws s3 mb s3://abdullahi-fastlane-certificates

$ export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxx

$ export AWS_ACCESS_KEY_ID=xxxxxxxxxxxx

$ export MATCH_PASSWORD=xxxxxxxxxxx

Listing 8-3

Creating S3 bucket and configuring environment variables

If a default region is not configured for your AWS CLI, specify the AWS region in the CLI command using flag --region.

Using the S3 bucket created, and the AWS credentials environment variable configured in Listing 8-3, the Matchfile created from Listing 8-2 is customized. An example content of the Matchfile is as shown in Listing 8-4.

$ cat Matchfile

s3_bucket("abdullahi-fastlane-certificates")

storage_mode("s3")

s3_access_key(ENV["AWS_ACCESS_KEY_ID"])

s3_secret_access_key(ENV["AWS_SECRET_ACCESS_KEY"])

type("development") # The default type, can be: appstore, adhoc, enterprise or development

app_identifier(["com.abdullahiolaoye.sampleapplication"])

username("xxxxxxxxx@email.com") # Your Apple Developer Portal username

team_id("XXXXXXXXXX") # Your team ID

# For all available options run `fastlane match --help`

# Remove the # in the beginning of the line to enable the other options

# The docs are available on https://docs.fastlane.tools/actions/match

Listing 8-4

New configured Matchfile

To use this example Matchfile, replace the content of your Matchfile with this content and change the S3 bucket name, app identifier, Apple username, and team ID to yours.

As shown in Listing 8-4, in addition to the S3 bucket name and AWS credentials environment variables, the app identifier, Apple developer username, and team ID are also configured in the Matchfile .

Storing Certificates and Keys in Amazon S3

With the Matchfile configured, now we can generate certificates and store in the centralized S3 bucket. Listing 8-5 shows an example of how to generate certificates for development and example output. The command in Listing 8-5 generates a lot of output so only a snippet is shown.

The command must be executed from the project directory where the Matchfile is stored.

User inputs such as Apple ID password and keychain password will be required for the command in Listing 8-5.

Listing 8-5. Fastlane generating credentials for development

When this command completes, the signing credentials will be stored in the mentioned S3 bucket, as seen in Figure 8-1.

Figure 8-1

Code signing credentials stored in S3 bucket per team ID

The signing credentials are organized with the team ID, within each team ID folder, the signing credentials are stored as shown in Figure 8-2.

Figure 8-2

Signing credentials for a specific team ID

Storing match Secrets and Parameters in AWS Secrets Manager

As mentioned earlier, the key benefit of using Fastlane match for code signing is the ease of reuse of one signing credential across different developers and build systems. With the signing credentials now stored in a centralized S3 bucket, credentials that will be used to properly access and use these stored code signing credentials must also be stored in a centralized location so they can be easily retrieved when needed.

So far, the credentials that will be required by other users of these signing credentials are the AWS credentials to access the centralized S3 bucket and the match password for decrypting the code signing credentials, all of which were configured in Listing 8-3. Additional information that should be stored are the S3 bucket name, app identifier, apple username, and team ID.

To store these, we will use AWS Secrets Manager, which is an AWS service designed for storing secrets. Figure 8-3 shows how to create a secret in AWS Secrets Manager to store these secrets.

Figure 8-3

Creating AWS Secrets Manager secret to store Fastlane secrets

After secret creation, it should be visible on the list of secrets. Figure 8-4 shows a successful secret creation.

Figure 8-4

Verifying secret creation

Next, I store the secrets, and it should be visible as shown in Figure 8-5. Now the secrets can either be retrieved manually or programmatically as part of a build process.

The secret keys used in Figure 8-5 are examples and can be customized to a key name that suits you. In later sections, I will show where these secret keys will be used in build scripts for automatic secret retrieval.

Figure 8-5

All secrets and parameters stored in AWS Secrets Manager

Setting Up Jenkins Environment

To use Jenkins and Fastlane for building and releasing iOS applications, some tools are required to be installed in the Jenkins environment. The required tools are verified in this section.

Setting Up and Verifying Xcode Installation

For any iOS application build, Xcode must be installed. When Xcode is installed from the App Store, it also installs its command line interface (CLI) tools which are used by Fastlane.

After Xcode installation, ensure to accept the license agreement and verify Xcode is launched successfully as shown in Figure 8-6.

Figure 8-6

Verifying Xcode installation

Also verify that Xcode command line tools are installed as shown in Listing 8-6.

$ xcode-select --version

xcode-select --version

xcode-select version 2384.

$ xcodebuild -version

xcodebuild -version

Xcode 13.0

Build version 13A233

$ xcodebuild

Command line invocation:

    /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild

User defaults from command line:

    IDEPackageSupportUseBuiltinSCM = YES

xcodebuild: error: The directory /Users/userA does not contain an Xcode project.

Listing 8-6

Verifying Xcode CLI tools

Verifying Fastlane Installation

Verify that Fastlane is installed in the Jenkins environment as shown in Listing 8-7.

Listing 8-7. Verifying Fastlane installation

Setting Up AWS CLI

AWS CLI will be used to retrieve all Fastlane secrets stored in AWS Secret Manager during Jenkins build so AWS CLI installation and configuration is required in the Jenkins environment.

AWS CLI can be verified and configured as shown in Listing 8-8.

$ aws –-version

$ aws configure

Listing 8-8

Verifying and configuring AWS CLI

Installing JSON Query Language (jq)

When secrets are retrieved from AWS Secrets Manager, they are returned in JSON and need to be parsed to retrieve the relevant string. For parsing, jq is used, hence it should be installed in the Jenkins environment. It can be installed with Homebrew, as shown in Listing 8-9.

$ brew install jq

Listing 8-9

Verifying jq installation

Automating Test and Build with Fastlane

In this section we will focus on how to automate testing and building iOS apps with Fastlane and Jenkins.

Adding Lane to Fastfile

First to add Fastlane to an iOS application project, a fastlane directory should be created in the root folder of the iOS project and within the Fastlane folder a file named Fastfile should be created, as shown in Listing 8-10.

$ mkdir fastlane

$ cd fastlane && touch Fastfile

Listing 8-10

Creating Fastlane directory and Fastfile from project root folder

The Fastfile is written in Ruby and contains all the instructions that Fastlane needs to perform action on an iOS application. Instructions within a Fastfile are written in lanes. A lane contains all the logic to perform a specific action on an iOS application, when a particular set of instructions are to be called, the name of the lane is passed to the Fastlane command.

Example lanes to test and build that can be added to the newly created Fastfile from Listing 8-10 are shown in Listing 8-11.

lane :build do

    match(

      type: "appstore",

      storage_mode: "s3",

      s3_bucket: ENV["S3_BUCKET"],

      s3_access_key: ENV["AWS_ACCESS_KEY_ID"],

      s3_secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],

      app_identifier: ENV["APP_IDENTIFIER"],

      username: ENV["APPLE_DEVELOPER_USERNAME"],

      team_id: ENV["TEAM_ID"]

    )

    increment_build_number(build_number: ENV["BUILD_ID"])

    gym(project: "SampleApp.xcodeproj")

  end

lane :test do

    scan(project: "SampleApp.xcodeproj",

              devices: ["iPhone Xs"])

  end

Listing 8-11

Adding Fastlane lanes for test and build to Fastfile

As seen, there are two lanes, namely, test and build. In the lane configurations, all the parameters stored in AWS Secrets Manager will be retrieved and made available to Fastlane through environment variables at build time.

Configuring Jenkins Project

In the previous chapter, a Jenkins project was set up and integrated with AWS CodeCommit for automated build. This project will be extended to add iOS build and test logic to its build configuration. An example build script can be seen in Listing 8-12.

set +x

echo "Set fastlane variables..."

export LC_ALL=en_US.UTF-8

export LANG=en_US.UTF-8

echo "Retrieving Secrets from AWS Secret Manager..."

ACCESS_KEY_ID=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString --output text | jq -r .AWS_ACCESS_KEY_ID`

SECRET_ACCESS_KEY=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString --output text | jq -r .AWS_SECRET_ACCESS_KEY`

MATCH_PASSWORD=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString --output text | jq -r .MATCH_PASSWORD`

S3_BUCKET=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString --output text | jq -r .S3_BUCKET`

APP_IDENTIFIER=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString --output text | jq -r .APP_IDENTIFIER`

APPLE_DEVELOPER_USERNAME=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString --output text | jq -r .APPLE_DEVELOPER_USERNAME`

TEAM_ID=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString --output text | jq -r .TEAM_ID`

echo "Setting fastlane match environment variables..."

export AWS_ACCESS_KEY_ID=$ACCESS_KEY_ID && export AWS_SECRET_ACCESS_KEY=$SECRET_ACCESS_KEY && export MATCH_PASSWORD=$MATCH_PASSWORD && export S3_BUCKET=$S3_BUCKET && export APP_IDENTIFIER=$APP_IDENTIFIER && export APPLE_DEVELOPER_USERNAME=$APPLE_DEVELOPER_USERNAME && export TEAM_ID=$TEAM_ID

echo "Starting Fastlane..."

fastlane test

fastlane build

Listing 8-12

Example test and build script

As seen in Listing 8-12, first parameters are retrieved from AWS Secret Manager and configured as environment variable then the test and build lanes are called to test and build the iOS app. The script is added to Jenkins to overwrite the existing build script, as shown in Figure 8-7.

Figure 8-7

Jenkins build configuration

Triggering Jenkins Project

With the Fastlane build and test lanes added to the iOS project and the Jenkins build configured accordingly, a new commit can then be created in AWS CodeCommit to start a new build. Code can be committed and pushed, as shown in Listing 8-13.

$ git add . && git commit -m "Build and test app"

[master ea9e896] Build and test app

 1 file changed, 1 insertion(+), 33 deletions(-)

$ git push

Enumerating objects: 7, done.

Counting objects: 100% (7/7), done.

Delta compression using up to 6 threads

Compressing objects: 100% (4/4), done.

Writing objects: 100% (4/4), 368 bytes | 368.00 KiB/s, done.

Total 4 (delta 2), reused 0 (delta 0), pack-reused 0

To codecommit::us-east-1://devops-ios-repository

   fb68e27..ea9e896  master -> master

Listing 8-13

Committing and pushing changes to AWS CodeCommit

This kicks off a build on Jenkins, as shown in Figure 8-8.

Figure 8-8

New commit to kickoff iOS app test and build

The build logs can be seen in Jenkins, but a snippet of the log for the build triggered is shown in Listing 8-14.

Listing 8-14. Fastlane test and build Jenkins log snippet

Jenkins must be able to find the required installed tools such as aws and jq. If you get build errors such as “jq: command not found” or “aws: command not found,” configure Jenkins PATH variable appropriately or specify the full path, which includes their installation directory in the build script, for example, /usr/local/bin/aws.

Automating Release to App Store Connect

App Store Connect allows you to manage your iOS applications. From there, iOS apps can be distributed for user testing using TestFlight and releasing apps to the App Store. To manage an iOS app on App Store Connect, the built application is first uploaded to the App Store Connect portal and then managed from there.

Fastlane also automates the release of iOS apps to App Store Connect. In this section, we will extend the Jenkins project configured in the previous section that tests and builds applications to add an action to release the built app to App Store Connect.

We will investigate two use cases of App Store Connect, releasing an app to TestFlight for beta testing and releasing an app that will be submitted to the App Store.

Setting Up App Store Connect API Key

For Fastlane to interact with App Store Connect programmatically without manual intervention, an App Store Connect API key would be generated for authenticating Fastlane. To generate an API key, you start from the App Store Connect dashboard, access the Users and Access page, as shown in Figure 8-9.

Figure 8-9

App Store Connect dashboard

This page lists all the users that exist for this Apple developer account and their roles. Switch to the Keys tab as shown in Figure 8-10.

Figure 8-10

Accessing the Keys tab from the Users and Access page

If this is the first time generating an API key for App Store Connect API, access must be requested first, as shown in Figure 8-11.

Figure 8-11

Request access to App Store Connect API

Once the access request is approved (usually instantly), an API key can be generated, as shown in Figure 8-12.

Figure 8-12

Generating App Store Connect API key

Provide a name and access level for the API key. An example is shown in Figure 8-13.

Figure 8-13

Providing API key name and generating it

Once the API key is generated, it becomes visible for download. Download the API key, as shown in Figure 8-14.

This key can only be downloaded once so it should be stored in a safe location. If lost, it must be regenerated again.

Figure 8-14

Downloading API key

After downloading the API key, the option to download is no longer available, as shown in Figure 8-15.

Figure 8-15

Download API key option not available

The API key is downloaded in a p8 extension file. To use the API key in consequent steps, it should be base64 encoded, as shown in Listing 8-15.

$ base64 AuthKey_AxxxxxxxBY.p8 > base64output.txt

Listing 8-15

Base64 encode App Store Connect API Key

In addition to the API key downloaded and base64 encoded, the Issuer ID and the Key ID should also be retrieved and stored in a safe location.

Storing Secrets in AWS Secret Manager

All the secrets retrieved from the App Store Connect portal will be used by the build process so there is a need to store these in a centralized secure location so they can be retrieved at build time.

For this use case, AWS Secrets Manager will be used to store the App Store Connect secrets similarly to how it is used to store Fastlane secrets and parameters. A new secret is created as shown in Figure 8-16.

Figure 8-16

Creating secrets to store App Store Connect Secrets

The secrets are then stored as shown in Figure 8-17. The base64-encoded API key is stored as a value with key API_KEY. The key ID, and issuer ID retrieved from the App Store Connect portal are stored as values with keys KEY_ID and APP_STORE_CONNECT_ISSUER_ID, respectively.

Figure 8-17

Storing App Store Connect Secrets

TestFlight Release

TestFlight allows iOS developers to invite real users to test apps to collect feedback before they are released publicly on the App Store. To use TestFlight, the app is built and released to App Store Connect. Here we will see how app release to TestFlight can be automated with Fastlane.

Adding TestFlight Lane to Fastfile

Extending the previously created Fastfile where the test and build lanes are configured, a TestFlight lane can be added. Listing 8-16 shows an example lane configuration for TestFlight.

lane :testflight do

  api_key = app_store_connect_api_key(

    key_id: ENV["KEY_ID"],

    issuer_id: ENV["APP_STORE_CONNECT_ISSUER_ID"],

    key_content: ENV["API_KEY"],

    duration: 1200,

    in_house: false,

    is_key_content_base64: true,

  )

  pilot(

    api_key: api_key,

 )

  End

Listing 8-16

Fastlane lane for TestFlight release

As seen in Listing 8-16, the Key ID, Issuer ID, and API Key will be retrieved from AWS Secrets Manager and made available through environment variables for Fastlane consumption at runtime by the build process.

Configuring Jenkins Project

The Jenkins build script configured from Listing 8-12 should be modified to add the new TestFlight lane, as shown in Listing 8-17.

set +x

echo "Set fastlane variables..."

export LC_ALL=en_US.UTF-8

export LANG=en_US.UTF-8

echo "Retrieving Secrets from AWS Secret Manager..."

ACCESS_KEY_ID=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString --output text | jq -r .AWS_ACCESS_KEY_ID`

SECRET_ACCESS_KEY=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString --output text | jq -r .AWS_SECRET_ACCESS_KEY`

MATCH_PASSWORD=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString --output text | jq -r .MATCH_PASSWORD`

S3_BUCKET=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString --output text | jq -r .S3_BUCKET`

APP_IDENTIFIER=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString --output text | jq -r .APP_IDENTIFIER`

APPLE_DEVELOPER_USERNAME=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString --output text | jq -r .APPLE_DEVELOPER_USERNAME`

TEAM_ID=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString  --output text | jq -r .TEAM_ID`

APP_STORE_CONNECT_ISSUER_ID=`aws secretsmanager get-secret-value --secret-id app-store-connect-secret --query SecretString --output text | jq -r .APP_STORE_CONNECT_ISSUER_ID`

KEY_ID=`aws secretsmanager get-secret-value --secret-id app-store-connect-secret --query SecretString --output text | jq -r .KEY_ID`

API_KEY=`aws secretsmanager get-secret-value --secret-id app-store-connect-secret --query SecretString --output text | jq -r .API_KEY`

echo "Setting fastlane match environment variables..."

export AWS_ACCESS_KEY_ID=$ACCESS_KEY_ID && export AWS_SECRET_ACCESS_KEY=$SECRET_ACCESS_KEY && export MATCH_PASSWORD=$MATCH_PASSWORD && export S3_BUCKET=$S3_BUCKET && export APP_IDENTIFIER=$APP_IDENTIFIER && export APPLE_DEVELOPER_USERNAME=$APPLE_DEVELOPER_USERNAME && export TEAM_ID=$TEAM_ID

echo "Setting app store connect secret environemnt variables"

export APP_STORE_CONNECT_ISSUER_ID=$APP_STORE_CONNECT_ISSUER_ID && export KEY_ID=$KEY_ID && export API_KEY=$API_KEY

echo "Starting Fastlane..."

fastlane test

fastlane build

fastlane testflight

Listing 8-17

Adding TestFlight to Jenkins build script

The App Store Connect secrets stored in AWS Secrets Manager are retrieved, set as environment variables before the testflight lane is called.

Triggering Jenkins Project

The TestFlight lane has been added to the application source code Fastfile and the Jenkins project modified accordingly, now the Jenkins project can be triggered by creating a new commit in AWS CodeCommit, as shown in Listing 8-18.

$ git add . && git commit -m "Upload to testflight"

[master 3a1febb] Upload to testflight

 1 file changed, 17 insertions(+), 1 deletion(-)

$ git push

Enumerating objects: 7, done.

Counting objects: 100% (7/7), done.

Delta compression using up to 6 threads

Compressing objects: 100% (4/4), done.

Writing objects: 100% (4/4), 580 bytes | 580.00 KiB/s, done.

Total 4 (delta 2), reused 0 (delta 0), pack-reused 0

To codecommit::us-east-1://devops-ios-repository

   ea9e896..3a1febb  master -> master

Listing 8-18

Creating a new commit in CodeCommit for TestFlight

The new commit is visible on Jenkins as shown in Figure 8-18.

Figure 8-18

New commit to trigger TestFlight release

Snippet of the log for the build triggered is shown in Listing 8-19.

Listing 8-19. Snippet of Jenkins logs for TestFlight upload

From the TestFlight console on App Store Connect, the just uploaded build is visible, as shown in Figure 8-19.

Figure 8-19

Fastlane TestFlight upload

Automating Release to the App Store

If an app is to be uploaded to App Store Connect for App Store submission, this process can also be automated by Fastlane.

Adding an App Store Lane to Fastfile

The lane configuration for the App Store should also be added to the same Fastfile used in the previous actions. Listing 8-20 shows an example lane configuration for App Store upload.

lane :appstore do

  api_key = app_store_connect_api_key(

    key_id: ENV["KEY_ID"],

    issuer_id: ENV["APP_STORE_CONNECT_ISSUER_ID"],

    key_content: ENV["API_KEY"],

    duration: 1200,

    in_house: false,

    is_key_content_base64: true,

  )

  deliver(

    api_key: api_key,

    force: true,

    run_precheck_before_submit: false

  )

  End

Listing 8-20

App Store lane configuration

Similarly, to other lane configurations, credentials are retrieved from build time environment variables.

Configuring Jenkins Project

The Jenkins build configuration for App Store upload is very similar to that of the TestFlight build because both require an upload to App Store Connect. The only difference is that the appstore lane is called by Fastlane and not the testflight lane, as shown in Listing 8-17. Listing 8-21 shows the Fastlane section of the build script defined in Listing 8-17 and shows the minor change that should be made to the build script.

echo "Starting Fastlane..."

fastlane test

fastlane build

fastlane appstore

Listing 8-21

Minor changes required for App Store lane

Triggering Jenkins Project

With the appstore lane added to Fastfile and Jenkins build script configured accordingly, a new commit can be created in CodeCommit, as shown in Listing 8-22.

$ git add . && git commit -m "Upload to appstore"

[master 83fa283] Upload to appstore

 1 file changed, 13 insertions(+), 13 deletions(-)

$ git push

Enumerating objects: 7, done.

Counting objects: 100% (7/7), done.

Delta compression using up to 6 threads

Compressing objects: 100% (4/4), done.

Writing objects: 100% (4/4), 469 bytes | 469.00 KiB/s, done.

Total 4 (delta 2), reused 0 (delta 0), pack-reused 0

To codecommit::us-east-1://devops-ios-repository

   0f80b39..83fa283  master -> master

Listing 8-22

Creating new commit for App Store upload

This starts a build on Jenkins; a snippet of the build logs for App Store upload is shown in Listing 8-23.

Listing 8-23. Snippet of Jenkins logs for App Store upload

The uploaded build is visible on App Store Connect, as shown in Figure 8-20.

Figure 8-20

Build uploaded for App Store release

Summary

Fastlane is a powerful tool that helps to automate a lot of otherwise manual tasks in iOS app development, and integrating this tool with Jenkins has opened up more automation possibilities. In this chapter we explored some of those possibilities and investigated how we can use Jenkins automated builds to automate the test, build, and release of iOS apps to App Store Connect.

The testing that was covered in this chapter with Fastlane uses iOS simulator for testing. Next, we will dive deep into testing iOS applications on the AWS cloud on real devices using AWS Device Farm.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

A. OlaoyeBeginning DevOps on AWS for iOS Developmenthttps://doi.org/10.1007/978-1-4842-8023-2_9

9. Testing with AWS Device Farm

Abdullahi Olaoye1  

(1)

McKinney, TX, USA

When iOS applications are tested on iOS simulators, you can test functionality and behavior of the app, but the simulator doesn’t work exactly like a real hardware device, hence an app may work fine on a simulator but behave differently when installed on a real iOS device. Testing your iOS application on real mobile devices during development improves quality of the application because you can simulate how real users will interact with the application.

In this chapter, we will explore how testing application on real mobile devices works on AWS Device Farm and integrate AWS Device Farm with Jenkins to automate the development and testing process.

All example code used in this chapter can be found at https://github.com/abdulola/devops-on-aws-ios-development/tree/main/Chapter_9.

Introduction to AWS Device Farm

AWS Device Farm is a service for testing and interacting with iOS, Android, and web applications on real physical devices such as phones and tablets that are hosted on the AWS Cloud. With Device Farm, you can set up automated testing for your application with a variety of test frameworks supported and set up remote access to devices where you can install your app and interact with the application in real time. However, we will be focusing on the automated testing feature of AWS Device Farm.

AWS Device Farm has some terminologies which we will be using as we explore the service. Here are some of those terminologies and what they mean:

Test – An individual test case.

Suite – A group of tests organized in a test package, containing one or more tests.

Job – The test of a single app against a single device with one or more suites.

Run – A specific build of an app with a set of tests to be run on a set of devices. A run consists of one or more jobs.

Project – A logical workspace that contains runs. Each run is for a single test of a single app against one or more devices.

Device Pool – A collection of devices that share similar characteristics

To test an iOS application on AWS Device Farm, you provide the ipa application package generated from the application build, you also provide your test use cases or use the build-in test framework, select the physical devices to use for testing, and configure the state of the device the app will be tested on. After the test, AWS Device Farm generates a report; some of the components of the report include

Pass and fail information

Crash reports

Test and device logs

Screenshots

Performance data

These reports are retained on AWS Device Farm for 400 days.

Generating ipa Package for App Testing

A testable iOS application archive file (ipa) for an app is required to schedule a test run on AWS Device Farm, the ipa that will be used for testing has to be one that was specifically built for testing.

A testing application package can be generated from Xcode. An application can be built for testing, as shown in Figure 9-1.

Figure 9-1

Building iOS application for testing

Once the build is completed, an application file is generated that can be exported and used for testing. Figure 9-2 shows how to access the directory location where the generated application archive is stored.

Figure 9-2

Opening build artifact location

The application archive (with .app extension) should be retrieved as shown in Figure 9-3.

Figure 9-3

Retrieving application archive for testing

An ipa archive can be created for Device Farm testing from the application archive retrieved from Figure 9-3, as seen in Listing 9-1.

The commands in Listing 9-1 must be executed from the Debug-iphoneos directory shown in Figure 9-3.

$ mkdir Payload && cp -r SampleApp.app Payload/

$ zip -r Payload.zip Payload && mv Payload.zip SampleApp.ipa

Listing 9-1

Generating ipa for Device Farm testing

As shown in Listing 9-1, a Payload directory is created, and the application file is copied into the directory. The directory is then archived and renamed with an ipa extension.

Setting Up a Device Farm Project and Scheduling Test Runs

With the application ipa file generated for Device Farm testing, we will explore how to schedule test runs with AWS Device Farm through the AWS Management Console and the AWS CLI.

Scheduling Test Runs with AWS Console

The AWS Device Farm Console can be accessed from the AWS Management Console as shown in Figure 9-4.

Figure 9-4

Accessing AWS Device Farm Console

AWS Device Farm is only currently available in the us-west-2 AWS region.

Tests in AWS Device Farm are run within a project; an existing project can be used, or a new project can be created as shown in Figure 9-5.

Figure 9-5

Creating a Device Farm Project

AWS Device Farm supports a variety of testing framework and built-in test types. Currently for iOS, Device Farm supports the following testing frameworks:

Appium

Calabash

UI Automation

XCTest

XCTest UI

For built-in test types that don’t require you to write or maintain test scripts, it offers Built-in Fuzz.

We will explore how to run tests in AWS Device Farm with the Built-in Fuzz test provided and XCTest UI testing framework.

Running Built-In Fuzz Test

The Built-in Fuzz test randomly sends user interface events to devices and then reports results. From within a Device Farm project, an automated test run can be created, as shown in Figure 9-6.

Figure 9-6

Creating an automated test run

The application ipa archive can be uploaded as shown in Figure 9-7.

Figure 9-7

Uploading application ipa

After the application upload, some details of the application are displayed, and the name of the run can also be configured as shown in Figure 9-8.

By default, the run name is set as the name of the ipa application uploaded. This name can be changed.

Figure 9-8

Configuring run name

Next, we configure the test by selecting the type of test framework to use. Figure 9-9 shows the selection of the Built-in Fuzz framework from the available options. All the default Fuzz configuration can be left as is.

Figure 9-9

Selecting Built-in Fuzz test framework

After providing application archive and configuring test framework, you select the type of physical devices to test the application on. As seen in Figure 9-10, you can use an already configured device pool or create your own customized device pool. The device pool used in Figure 9-10 is Top Devices; it also shows that the app provided is only compatible with two of these devices.

Figure 9-10

Selecting device pool

You can also configure the device state. Device state configuration allows you to select the device language, and configure network options such as Wi-Fi, Bluetooth, GPS, etc., for applications that depend on these device functions. These are left as default for this sample app, as shown in Figure 9-11.

Figure 9-11

Reviewing device state configuration and starting run

Once a run is started, it goes into a pending state. It takes a few minutes for the run to start depending on the availability of physical devices.

To see more details and monitor tests running on each of the devices in the device pool, you can select the test and the run details are displayed as shown in Figure 9-12.

Figure 9-12

Details of run in pending state

When a run completes, the status of the run can be viewed. The number of tests executed, and their different status are shown. A pie chart also shows the percentage pass and other states (if applicable), as shown in Figure 9-13.

Figure 9-13

Test status for run

To view test report details for a particular device in the device pool, select the device. From Figure 9-13, report can be viewed for iPhone 11 and iPhone XR. An example report for iPhone 11 can be seen in Figure 9-14.

Figure 9-14

Test report shown for iPhone 11

Running XCTest UI Test

The XCTest UI framework allows you to test UI capabilities for your app. The test bundle is produced by Xcode when you build your project for testing. To use the XCTest UI framework on Device Farm, the same app package generated from Listing 9-1 can be used but the test bundle must be retrieved and formatted into an ipa file as shown next.

Generating Test Bundle for XCTest UI

The test bundle is saved in the same location as the build artifact and can be accessed as shown in Figure 9-2. The test bundle has the naming pattern of <App Name>UITests-Runner.app. The test bundle for the sample in Figure 9-1 can be retrieved, as shown in Figure 9-15.

Figure 9-15

Retrieving XCTests UI bundle for testing

An ipa archive can be created for Device Farm compatibility from the test bundle retrieved from Figure 9-15, as seen in Listing 9-2.

$ mkdir Payload && cp -r SampleAppUITests-Runner.app Payload/

$ zip -r Payload.zip Payload && mv Payload.zip XCTestUI.ipa

Listing 9-2

Generating ipa for Device Farm testing

Like the application build ipa generated earlier, in Listing 9-2, a Payload directory is created, and the test bundle is copied into the directory. The directory is then archived and renamed with an ipa extension.

Uploading Application and Test package

To start another test run, a test run creation can be started as shown in Figure 9-6 and an app ipa package can be uploaded as shown in Figure 9-7. The default test run name can be modified as shown in Figure 9-16.

Figure 9-16

Customizing test run name

For the test framework, XCTest UI should be selected from the options. Once the test type is selected, the option to upload the test ipa package created from Listing 9-2 also appears as shown in Figure 9-17.

Figure 9-17

Selecting test framework and uploading test package

Once the test package is uploaded, select the execution environment for the test. A custom environment should be selected and the YAML test spec can be configured as shown in Figure 9-18.

Figure 9-18

Configuring execution environment and test specification

The test spec contains the different phases and the commands that will be executed to perform this test. The default YAML spec can be used as is, but it can also be modified to add custom commands.

Creating a Custom Device Pool

The next step is to select a device pool to run the test. In Figure 9-10, a managed device pool was used to run the test, but let’s explore how we can create a custom device pool. Creating a custom device pool provides you the control to select the exact devices you want to run tests on.

A device pool can be created as shown in Figure 9-19.

Figure 9-19

Creating a custom device pool

Creating a custom device pool consists of providing a name for the device pool and selecting the mobile devices that should be in the pool. Figure 9-20 shows an example static device pool creation with only one device, Apple iPhone X with iOS 14.6. With a dynamic device pool, you can create rules such that devices that matches the rule will be added to your device pool.

Figure 9-20

Custom device pool with Apple iPhone X

Running a Test

The custom device pool created can be selected for testing as shown in Figure 9-21.

Figure 9-21

Selecting custom device pool

As shown in Figure 9-21, there is only one device in this device pool and this device is compatible with the application being tested. The rest of the configuration can be left as default and run started as shown in Figure 9-11. Once the test run is created, its status can be monitored similarly to Figure 9-12. Figure 9-22 shows an example of the completed XCTest UI Test.

Figure 9-22

Completed XCTest UI test

Scheduling Test Runs with AWS CLI

Just as shown with the AWS Console, you first create a Device Farm project. Creating a project and example output is shown in Listing 9-3.

$ aws devicefarm create-project --name SampleApp2 --region us-west-2

{

    "project": {

        "arn": "arn:aws:devicefarm:us-west-2:123456789101:project:abcdefg2-46a7-4ebf-b3ab-7de683ad43cd",

        "name": "SampleApp2",

        "created": "2090-20-20T16:25:11.049000+00:00"

    }

}

Listing 9-3

Creating Device Farm project with AWS CLI

The project created with the AWS CLI can also be seen on the AWS Console, as shown in Figure 9-23.

The Amazon Resource Name (ARN) returned from the command should be stored and will be used in consequent CLI commands for this project.

Figure 9-23

Project created with the CLI viewed on AWS Console

The process to upload the application to AWS Device Farm through the AWS CLI is in two steps. First you create an upload, and this will generate an S3 presigned URL where the file will be uploaded to. Listing 9-4 shows how to create an upload for the project created in Listing 9-3 and shown in Figure 9-23.

$ aws devicefarm create-upload --project-arn arn:aws:devicefarm:us-west-2:123456789101:project:abcdefg2-46a7-4ebf-b3ab-7de683ad43cd --name SampleApp.ipa --type IOS_APP --region us-west-2

{

    "upload": {

        "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:abcdefg2-46a7-4ebf-b3ab-7de683ad43cd/0057dee1-a704-43f6-83dd-a993d0ffd4f8",

        "name": "SampleApp.ipa",

        "created": "2090-20-20T16:33:23.578000+00:00",

        "type": "IOS_APP",

        "status": "INITIALIZED",

        "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A123456789101%3Aproject%3Aabcdefg2-46a7-4ebf-b3ab-7de683ad43cd/uploads/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A123456789101%3Aupload%3Aabcdefg2-46a7-4ebf-b3ab-7de683ad43cd/0057dee1-a704-43f6-83dd-a36575959ffd4f8/SampleApp.ipa?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20211017T163323Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20211017%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=4758748956745646fhfgh68768984292",

        "category": "PRIVATE"

    }

}

Listing 9-4

Creating an upload for project and output

Store the upload ARN returned from the command in the arn section as shown here. It will be used in consequent CLI commands against this upload.

The application can then be uploaded to the S3 signed URL returned when upload was created, as shown in Listing 9-5.

$ curl -T SampleApp.ipa https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A123456789101%3Aproject%3Aabcdefg2-46a7-4ebf-b3ab-7de683ad43cd/uploads/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A123456789101%3Aupload%3Aabcdefg2-46a7-4ebf-b3ab-7de683ad43cd/0057dee1-a704-43f6-83dd-a36575959ffd4f8/SampleApp.ipa?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20211017T163323Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20211017%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=4758748956745646fhfgh68768984292

Listing 9-5

Uploading application archive

The status of the upload can be retrieved with the AWS CLI, as shown in Listing 9-6. The status must be SUCCEEDED before a run can be scheduled using that application.

$ aws devicefarm get-upload --arn arn:aws:devicefarm:us-west-2:123456789101:upload:abcdefg2-46a7-4ebf-b3ab-7de683ad43cd/0057dee1-a704-43f6-83dd-a993d0ffd4f8 --region us-west-2

{

    "upload": {

        "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:abcdefg2-46a7-4ebf-b3ab-7de683ad43cd/0057dee1-a704-43f6-83dd-a993d0ffd4f8",

        "name": "SampleApp.ipa",

        "created": "2090-20-20T16:33:23.578000+00:00",

        "type": "IOS_APP ",

        "status": "SUCCEEDED",

        "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A123456789101%3Aproject%3Aabcdefg2-46a7-4ebf-b3ab-7de683ad43cd/uploads/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A123456789101%3Aupload%3Aabcdefg2-46a7-4ebf-b3ab-7de683ad43cd/0057dee1-a704-43f6-83dd-a993d0ffd4f8/SampleApp.ipa?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20211017T163908Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20211017%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=02d8120348419a7007602011f6786c80f53e930ce63c9f6ade13c43e41453530",

        "metadata": "{\"activity_name\":\"\",\"minimum_arm\":6,\"error_type\":null,\"package_name\":\"com.abdullahiolaoye.sampleapplication\",\"sdk_version\":1400,\"files\":{},\"warning_type\":null,\"supported_os\":\"14.0\",\"executable\":\"SampleApp\",\"platform\":[\"iPhoneOS\"],\"form_factor\":[1,2]}",

        "category": "PRIVATE"

    }

}

Listing 9-6

Checking upload status

Upload ARN passed in the --arn argument can be retrieved as shown in Listing 9-4.

One resource needed before a run can be scheduled is the device pool. To use the inbuilt device pool, you can use the AWS CLI to see the device pools available for a project as shown in Listing 9-7.

$ aws devicefarm list-device-pools --arn arn:aws:devicefarm:us-west-2:123456789101:project:abcdefg2-46a7-4ebf-b3ab-7de683ad43cd --region us-west-2

{

    "devicePools": [

        {

            "arn": "arn:aws:devicefarm:us-west-2::devicepool:082d10e5-d7d7-48a5-ba5c-b33d66efa1f5",

            "name": "Top Devices",

            "description": "Top devices",

            "type": "CURATED",

            "rules": [

                {

                    "attribute": "ARN",

                    "operator": "IN",

                    "value": "[\"arn:aws:devicefarm:us-west-2::device:CE68825ABE5A4740B56F10111FD47844\",\"arn:aws:devicefarm:us-west-2::device:692AE103025C4C8B8682D22556957C83\",\"arn:aws:devicefarm:us-west-2::device:2CC987870E204BB18E7E36EF5F3C3F45\",\"arn:aws:devicefarm:us-west-2::device:8766BD1DD11744B0ACEA337A6D9BEBEF\",\"arn:aws:devicefarm:us-west-2::device:1F7E40CA59B6467F9443943E14F93848\",\"arn:aws:devicefarm:us-west-2::device:8B4E8F953E0F4460BBC2A18A504D8FC2\",\"arn:aws:devicefarm:us-west-2::device:23E8A18A1966455A8DC1E2ABF8A07F05\",\"arn:aws:devicefarm:us-west-2::device:19B1FD045303482BAE75EE14AFC29AA5\",\"arn:aws:devicefarm:us-west-2::device:57D25EA0789643648E8B4F922B9559AB\",\"arn:aws:devicefarm:us-west-2::device:2330880C98CE4AD287D354F6A58F8450\"]"

                }

            ]

        },

        {

            "arn": "arn:aws:devicefarm:us-west-2::devicepool:1c59cfd0-ee56-4443-b290-7a808d9fd885",

            "name": "Web Performance",

            "type": "CURATED",

            "rules": [

                {

                    "attribute": "ARN",

                    "operator": "IN",

                    "value": "[\"arn:aws:devicefarm:us-west-2::device:2832D5722BEF4FF2B04498ECC4C1C2F6\",\"arn:aws:devicefarm:us-west-2::device:4F74D943F7594EFF96957E238B3CA131\",\"arn:aws:devicefarm:us-west-2::device:5F20BBED05F74D6288D51236B0FB9895\",\"arn:aws:devicefarm:us-west-2::device:1F7E40CA59B6467F9443943E14F93848\",\"arn:aws:devicefarm:us-west-2::device:A7CCFD183C314CE2B44B0BB284AAEA0A\"]"

                }

            ]

        }

    ]

}

Listing 9-7

Showing available device pools and output snippet

The command output returns a list of device pools and their Amazon Resource Name (ARN).

Store the ARN of the device pool to use in scheduling the run. To use the built-in “Top Devices” pool, store the ARN of this device pool as seen in the output of the command shown in Listing 9-7.

Now all the resources required to schedule a run have been configured. A test run can be scheduled with the Built-in Fuzz test type using the AWS CLI, as shown in Listing 9-8.

$ aws devicefarm schedule-run --project-arn arn:aws:devicefarm:us-west-2:123456789101:project:abcdefg2-46a7-4ebf-b3ab-7de683ad43cd --app-arn arn:aws:devicefarm:us-west-2:123456789101:upload:abcdefg2-46a7-4ebf-b3ab-7de683ad43cd/0057dee1-a704-43f6-83dd-a993d0ffd4f8 --device-pool-arn arn:aws:devicefarm:us-west-2::devicepool:082d10e5-d7d7-48a5-ba5c-b33d66efa1f5 --name SampleApp2.ipa --test type=BUILTIN_FUZZ --region us-west-2

{

    "run": {

        "arn": "arn:aws:devicefarm:us-west-2:123456789101:run:abcdefg2-46a7-4ebf-b3ab-7de683ad43cd/4f324434-2652-45f3-b065-9961e235cfd4",

        "name": "SampleApp2.ipa",

        "type": "BUILTIN_FUZZ",

        "platform": "IOS_APP",

        "created": "2090-20-20T16:45:02.097000+00:00",

        "status": "SCHEDULING",

        "result": "PENDING",

        "started": "2090-20-20T16:45:02.097000+00:00",

        "counters": {

            "total": 0,

            "passed": 0,

            "failed": 0,

            "warned": 0,

            "errored": 0,

            "stopped": 0,

            "skipped": 0

        } ,

        "totalJobs": 2,

        "completedJobs": 0,

        "billingMethod": "METERED",

        "seed": 2134223132,

        "appUpload": "arn:aws:devicefarm:us-west-2:123456789101:upload:abcdefg2-46a7-4ebf-b3ab-7de683ad43cd/0057dee1-a704-43f6-83dd-a993d0ffd4f8",

        "eventCount": 700,

        "jobTimeoutMinutes": 150,

        "devicePoolArn": "arn:aws:devicefarm:us-west-2:123456789101:devicepool:abcdefg2-46a7-4ebf-b3ab-7de683ad43cd/082d10e5-d7d7-48a5-ba5c-b33d66efa1f5",

        "radios": {

            "wifi": true,

            "bluetooth": false,

            "nfc": true,

            "gps": true

        }

    }

}

Listing 9-8

Scheduling a test run with AWS CLI

The project arn is passed to --project-arn argument, the upload arn is passed to the --app-arn, and the device pool arn is passed to the --device-pool-arn argument.

The example run scheduled in Listing 9-8 uses the “Top Devices” device pool.

Retrieve the run ARN from the output of the command in Listing 9-8. This would be used to get the status of the run from the AWS CLI.

The status of the scheduled run can be viewed on the AWS CLI as shown in Listing 9-9.

$ aws devicefarm get-run --arn arn:aws:devicefarm:us-west-2:123456789101:run:abcdefg2-46a7-4ebf-b3ab-7de683ad43cd/4f324434-2652-45f3-b065-9961e235cfd4 --region us-west-2

{

    "run": {

        "arn": "arn:aws:devicefarm:us-west-2:123456789101:run:abcdefg2-46a7-4ebf-b3ab-7de683ad43cd/4f324434-2652-45f3-b065-9961e235cfd4",

        "name": "SampleApp2.ipa",

        "type": "BUILTIN_FUZZ",

        "platform": "IOS_APP",

        "created": "2090-20-20T16:45:02.097000+00:00",

        "status": "RUNNING",

        "result": "PENDING",

        "started": "2090-20-20T16:45:02.097000+00:00",

        "counters": {

            "total": 0,

            "passed": 0,

            "failed": 0,

            "warned": 0,

            "errored": 0,

            "stopped": 0,

            "skipped": 0

        } ,

        "totalJobs": 2,

        "completedJobs": 0,

        "billingMethod": "METERED",

        "deviceMinutes": {

            "total": 0.0,

            "metered": 0.0,

            "unmetered": 0.0

        },

        "parsingResultUrl": "https://prod-us-west-2-results.s3-us-west-2.amazonaws.com/123456789101/abcdefg2-46a7-4ebf-b3ab-7de683ad43cd/4f324434-2652-45f3-b065-9961e235cfd4/parsingResult.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20211017T164740Z&X-Amz-SignedHeaders=host&X-Amz-Expires=259200&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20211017%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=95e44a5eaf28a5ea66246a770b3fa85cc3b68e9b6f935e40720e809113094886",

        "seed": 2134223132,

        "appUpload": "arn:aws:devicefarm:us-west-2:123456789101:upload:abcdefg2-46a7-4ebf-b3ab-7de683ad43cd/0057dee1-a704-43f6-83dd-a993d0ffd4f8",

        "eventCount": 700,

        "jobTimeoutMinutes": 150,

        "devicePoolArn": "arn:aws:devicefarm:us-west-2:123456789101:devicepool:abcdefg2-46a7-4ebf-b3ab-7de683ad43cd/082d10e5-d7d7-48a5-ba5c-b33d66efa1f5",

        "radios": {

            "wifi": true,

            "bluetooth": false,

            "nfc": true,

            "gps": true

        }

    }

}

Listing 9-9

Checking status of scheduled run

The run scheduled on the AWS CLI can also be viewed on the AWS Console, as shown in Figure 9-24.

Figure 9-24

Run scheduled on AWS CLI viewed on AWS Console

Once the test run completes, the test report can be viewed on the console, as shown in Figure 9-25.

Figure 9-25

Test report for run scheduled on the AWS CLI

The report is the same as shown in Figure 9-13 because the same application is used for testing.

AWS Device Farm Jenkins Plugin

AWS Device Farm can be integrated with Jenkins jobs through its plugin. With this integration, Jenkins jobs that build iOS apps can send artifacts to AWS Device Farm post build for testing on real devices without manual intervention.

Installing Plugin

AWS Device Farm plugin can be installed with the Jenkins Web UI through the Manage Plugins option in Jenkins settings page as shown in Figure 9-26.

Figure 9-26

Access Jenkins plugins

The plugin can be searched and installed, as shown in Figure 9-27.

Figure 9-27

Installing AWS Device Farm Jenkins plugin

Configuring Plugin

The configuration of the plugin involves providing AWS credentials the plugin will use in interacting with AWS Device Farm in your AWS account. The plugin supports IAM user credentials or getting temporary credentials through an IAM role. Irrespective of the type of IAM identity used, AWS Device Farm permission is needed by the plugin to submit test runs. Listing 9-10 shows an example IAM policy that allows full Device Farm access.

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Sid": "DeviceFarmAll",

            "Effect": "Allow",

            "Action": [ "devicefarm:*" ],

            "Resource": [ "*" ]

        }

    ]

}

Listing 9-10

Full AWS Device Farm access

An AWS Device Farm managed IAM policy can also be used by attaching the managed policy to the IAM user or IAM role the plugin will use. Figure 9-28 shows AWSDeviceFarmFullAccess managed policy attached to an IAM user created for Device Farm.

Figure 9-28

IAM user configured with AWS Device Farm managed policy

Likewise, if an IAM role is used to configure the plugin, Figure 9-29 shows an IAM role using the AWSDeviceFarmFullAccess managed policy.

Figure 9-29

IAM role configured with AWS Device Farm managed policy

To configure the plugin, access the system configuration from the Jenkins settings page as shown in Figure 9-30.

Figure 9-30

Accessing plugin configuration

On the configuration page, scroll to the AWS Device Farm section. Figure 9-31 shows an example configuration of the plugin with IAM user credentials.

Figure 9-31

Configuring plugin with IAM user credentials

To use an IAM role with the plugin, the plugin assumes that the Jenkins global environment is configured with AWS credentials and these credentials will be used to assume the role.

Global AWS credentials on Jenkins can be in the form of an EC2 IAM role attached to the Jenkins instance (if Jenkins is hosted on EC2), AWS credentials environment variables configured on Jenkins instance, etc.

The Plugin IAM role must be configured to trust the IAM entity (User, Role) that the Jenkins environment uses. Listing 9-11 shows an example of the Plugin IAM role trust relationship configuration.

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Principal": {

        "AWS": "<IAM User or Role ARN for Jenkins Environment>"

      },

      "Action": "sts:AssumeRole"

    }

  ]

}

Listing 9-11

Trust relationship of the Plugin IAM role

The IAM credentials used in the Jenkins environment which will be used to assume to the Plugin IAM role must also have permission to assume the role. An example IAM policy is shown in Listing 9-12, which shows how to configure permission for the Jenkins environment IAM User or Role to assume the Plugin IAM role.

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Action": "sts:AssumeRole",

      "Resource": "<Role ARN of the Device Farm Plugin Role>"

    }

  ]

}

Listing 9-12

Configuring permission to assume Plugin IAM role

After ensuring all permission are configured to use an IAM role with the Device Farm plugin. Figure 9-32 shows an example of how to configure an IAM role for the Plugin.

Figure 9-32

Configure Device Farm plugin with IAM Role

Automating AWS Device Farm Tests with Jenkins

In this section, we will explore how to integrate AWS Device Farm with Jenkins project already set up for automated iOS app builds from the previous chapter. The addition of AWS Device Farm to the Jenkins project will automatically start Device Farm tests whenever an app is built by Fastlane.

Configuring Jenkins Project

Open the existing Jenkins project that has been configured to build iOS apps and replace the build script with the script shown in Listing 9-13.

set +x

export TEAM_ID=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString --output text | jq -r .TEAM_ID`

/usr/bin/xcodebuild build-for-testing -scheme SampleApp -destination generic/platform=iOS DEVELOPMENT_TEAM=$TEAM_ID s-allowProvisioningUpdates -derivedDataPath $WORKSPACE

mkdir Payload && cp -r $WORKSPACE/Build/Products/Debug-iphoneos/SampleApp.app Payload/

zip -r Payload.zip Payload && mv Payload.zip SampleApp.ipa

Listing 9-13

Jenkins build script for Device Farm testing

This build script retrieves the Apple team ID from AWS Secrets Manager and configures it as an environment variable, builds the application for testing with Xcode command line tool (xcodebuild), saves the build artifact into the Jenkins workspace directory, and creates a testable ipa file ready to be used for testing in AWS Device Farm.

Input your Jenkins workspace directory in the script provided in Listing 9-13 before use.

An example of the build script when configured on Jenkins is as shown in Figure 9-33.

Figure 9-33

Device Farm build script on Jenkins

The artifact created in the build script will be passed over to AWS Device Farm for testing so to configure AWS Device Farm, create a post build action as shown in Figure 9-34.

Figure 9-34

Jenkins post build actions

In the post build actions, select running of tests on AWS Device Farm as shown in Figure 9-35.

Figure 9-35

Post build action to run AWS Device Farm tests

The configuration can be provided as shown in Figure 9-36. The project to use for testing and the device pool should be selected from the dropdown, the path where the ipa will be stored in the Jenkins workspace after build should be provided in the application option.

Figure 9-36

Device Farm post build configuration on Jenkins project

The example in Figure 9-36 shows an ipa that will be named SampleApp.ipa after build and is stored in the root of the workspace (as configured in Listing 9-13).

The name of the test run that will created by Jenkins on AWS Device Farm can also be configured, by default, the build tag is used to name the test run created.

The test framework to run and the execution environment should also be selected. Figure 9-37 shows an example of using the Built-in Fuzz test framework and executing the test run in a custom environment.

For more information on test environments in AWS Device Farm, see https://docs.aws.amazon.com/devicefarm/latest/developerguide/test-environments.html.

Figure 9-37

Configuring test framework and execution environment

Triggering Jenkins Project

The automated build can be triggered on the Jenkins project by creating a commit in AWS CodeCommit. You can also manually start a build on Jenkins to tests the newly address post build action.

A manual Jenkins build can be started as shown in Figure 9-38.

Figure 9-38

Starting a manual Jenkins build

An example Jenkins build logs for Device Farm in post build can be seen in Listing 9-14. As seen, the ipa file is first uploaded to S3 before the test is scheduled. The run is also named using the Jenkins build tag.

[AWSDeviceFarm] Using Project 'SampleApp'

[AWSDeviceFarm] Using DevicePool 'Top Devices'

[AWSDeviceFarm] Using App 'SampleApp.ipa'

[AWSDeviceFarm] Archiving artifact 'SampleApp.ipa'

[AWSDeviceFarm] Uploading SampleApp.ipa to S3

[AWSDeviceFarm] Waiting for upload SampleApp.ipa to be ready (current status: INITIALIZED)

[AWSDeviceFarm] Upload SampleApp.ipa succeeded

[AWSDeviceFarm] Getting test to schedule.

[AWSDeviceFarm] Scheduling 'BUILTIN_FUZZ' run 'jenkins-TestProject-96'

[AWSDeviceFarm] View the BUILTIN_FUZZ run in the AWS Device Farm Console: https://console.aws.amazon.com/devicefarm/home?#/projects/aaaabbbcc-33ab-4d34-87c6-576uhfdyur4/runs/abfj598n-ecfb-44cb-9d30-5958fhmfjhd

[AWSDeviceFarm] Waiting for test run to complete.

Listing 9-14

Device Farm Jenkins build logs

Once the build is complete, the results can be seen on the Jenkins project homepage. The page shows the number of tests in Pass, Warn, Skip, Fail, Error, and Stop states. Figure 9-39 shows that for the recently executed test, all six tests passed.

Figure 9-39

Test status on Jenkins project page

A detailed test report can be reviewed on the Device Farm console. Figure 9-40 shows the report for the Jenkins test run in question.

Figure 9-40

Test report on AWS Device Farm console

Summary

Introducing AWS Device Farm testing into iOS application development allows you to simulate real-world environments to run tests and reproduce issues to improve quality and user experience. The plugin and APS available for AWS Device Farm also make it feasible to integrate it with development workflows. In this chapter we explored the key benefits of AWS device farm and covered how it can be integrated with iOS development workflow on Jenkins.

In the next chapter, we will investigate how all the main development components covered in this chapter and previous chapters can be orchestrated together as stages in Continuous Integration Continuous Delivery (CICD) pipelines.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

A. OlaoyeBeginning DevOps on AWS for iOS Developmenthttps://doi.org/10.1007/978-1-4842-8023-2_10

10. Continuous Delivery Pipelines for iOS App Development

Abdullahi Olaoye1  

(1)

McKinney, TX, USA

A Continuous Integration Continuous Delivery (CICD) pipeline is an automated definition of the steps that occur in software development right from the source and version control system to delivering the new software version to users and customers. In the previous chapters, I introduced DevOps concepts and tools to different aspects and lifecycle of an iOS app development, from building, testing, to delivery. In this chapter, I will demonstrate how CICD pipelines can help automate and orchestrate all these different components into one streamlined process.

While there are a variety of options available when it comes to CICD pipelines, I will be focusing on Jenkins Pipeline and AWS CodePipeline. I will demonstrate how these pipeline types can integrate with iOS development tools to accomplish CICD for iOS application development.

All example code used in this chapter can be found at https://github.com/abdulola/devops-on-aws-ios-development/tree/main/Chapter_10.

Jenkins Pipeline

According to the Jenkins website, “Jenkins Pipeline (or simply “Pipeline” with a capital “P”) is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.”

For a given project using Jenkins Pipeline for CICD, the Pipeline is defined as code in a text file called “Jenkinsfile” and this text file is expected to be part of the projects source control repository so the pipeline can be version-controlled like the rest of the application code. When writing a Pipeline Jenkinsfile, the different syntaxes supported are Declarative and Scripted. I will be covering the declarative syntax as it’s more recent and makes it easier to read and understand Jenkins Pipeline code. On the other hand, the scripted syntax follows Groovy language syntax.

As an alternative to defining syntax in a Jenkinsfile, Pipeline syntax can also be defined on the Jenkins UI but it is best practice to define your Pipeline syntax as code and store it with the rest of your application code for version control.

The first step to setting up a Jenkins Pipeline is to create the Pipeline on Jenkins. Pipeline creation can be initiated as shown in Figure 10-1.

Figure 10-1

Setting up Jenkins Pipeline

Avoid having spaces in the pipeline name as it is known to cause issues during pipeline invocation.

Like setting up a Jenkins project as shown in the previous chapters, you can configure build triggers for the Jenkins Pipeline with a CodeCommit repository, as shown in Figure 10-2.

Figure 10-2

Setting up build trigger for Pipeline

As shown in Figure 10-3, I am configuring the pipeline to get the pipeline script, that is, the Jenkinsfile from the source CodeCommit repository then configuring the repository and credentials.

Figure 10-3

Configuring Pipeline definition location

The exact location of the Pipeline script within the repository must also be defined. In this example, the Pipeline script location is configured as shown in Figure 10-4.

Figure 10-4

Configuring Pipeline script path

After configuring the Pipeline script path, Pipeline creation can be saved to complete creation, as shown in Figure 10-4.

Mostly, Jenkins Pipeline script file is named Jenkinsfile; the name can be changed to any name of your choice. Whatever name used should be configured as shown in Figure 10-4.

As referenced in Figure 10-4, a Jenkinsfile must be added to the source repository. Figure 10-5 shows a repository structure after a Jenkinsfile is added.

Figure 10-5

Repository structure with Jenkinsfile added

The sample application code shown in Figure 10-5 is the same used in the previous chapters.

Fastlane Test Stage

With Pipeline already set up on Jenkins, and Jenkinsfile added to the source repository, you can start adding stages to the pipeline that represents different steps in your iOS application development process by adding content to your Jenkinsfile. First let’s see how to add a stage to run Fastlane tests.

Listing 10-1 shows a Jenkinsfile that defines a single-stage Pipeline to run Fastlane test.

pipeline {

    agent any

    stages {

        stage("Run fastlane test") {

            steps{

                sh """

                export LC_ALL=en_US.UTF-8 && export LANG=en_US.UTF-8

                fastlane test

                """

            }

        }

    }

}

Listing 10-1

Single-stage Pipeline definition to run Fastlane test

When these changes are committed and pushed to CodeCommit, the Pipeline triggers and it reads the Jenkinsfile in the repository to define steps in the pipeline. A successful execution of the pipeline defined in Listing 10-1 is shown in Figure 10-6.

Figure 10-6

Single-stage pipeline execution for Fastlane test

Jenkins must be able to find Fastlane. If you get errors that Fastlane cannot be found, configure Jenkins PATH variable appropriately or specify the full path to the Fastlane installation directory in the Jenkinsfile.

In addition, the pipeline logs can be retrieved to see more details on the execution and the steps. Logs snippet for the pipeline execution shown in Figure 10-6 is given in Listing 10-2.

Started by remote host SQSTrigger with note: User invoked: arn:aws:iam::123456789106:user/abdullahi2

Obtained jenkins/Jenkinsfile from git https://git-codecommit.us-east-1.amazonaws.com/v1/repos/devops-ios-repository

Running in Durability level: MAX_SURVIVABILITY

[Pipeline] Start of Pipeline

[Pipeline] node

.

.

.

.

[Pipeline] // stage

[Pipeline] withEnv

[Pipeline] {

[Pipeline] stage

[Pipeline] { (Run fastlane test)

[Pipeline] sh

+ export LC_ALL=en_US.UTF-8

+ LC_ALL=en_US.UTF-8

+ export LANG=en_US.UTF-8

+ LANG=en_US.UTF-8

+ fastlane test

.

.

.

.

[Pipeline] }

[Pipeline] // stage

[Pipeline] }

[Pipeline] // withEnv

[Pipeline] }

[Pipeline] // node

[Pipeline] End of Pipeline

Finished: SUCCESS

Listing 10-2

Jenkins Pipeline execution logs

AWS Device Farm Test Stage

To add a Device Farm stage to Pipeline, the AWS Device Farm Jenkins plugin must be installed. In the previous chapter, I demonstrated how to install the plugin and interact with it using a Jenkins freestyle project. But here we will be interacting with the plugin through Pipeline, so let’s generate the accurate syntax to do that. You can access the Pipeline syntax generator from within the Jenkins Pipeline as shown in Figure 10-7.

Figure 10-7

Access Pipeline syntax generator

Select the AWS Device Farm step from the dropdown of options as shown in Figure 10-8.

Figure 10-8

Selecting AWS Device Farm step

Enter the Device Farm test configuration as shown in Figures 9-36 and 9-37 (from Chapter 9), and generate Pipeline script for the configuration entered as shown in Figure 10-9.

Figure 10-9

Generating Pipeline script for AWS Device Farm step

The Pipeline script generated should be copied and will be used in configuring the Device Farm stage in Pipeline Jenkinsfile as shown in Listing 10-3, which shows appending the configuration of a Device Farm stage to the existing sample app Jenkinsfile.

pipeline {

    agent any

    environment {

        TEAM_ID = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString ​--output text | jq -r .TEAM_ID'

        ).trim()}"""

    }

    stages {

        stage("Run fastlane test") {

            steps{

                sh """

                export LC_ALL=en_US.UTF-8 && export LANG=en_US.UTF-8

                fastlane test

                """

            }

        }

        stage("Run AWS Device Farm test") {

            steps {

                sh '/usr/bin/xcodebuild build-for-testing ​-scheme SampleApp -destination generic/platform=iOS DEVELOPMENT_TEAM=${TEAM_ID} ​-allowProvisioningUpdates -derivedDataPath ${WORKSPACE}'

                sh 'mkdir Payload && cp -r ${WORKSPACE}/Build/Products/Debug-iphoneos/SampleApp.app Payload/'

                'zip -r Payload.zip Payload && mv Payload.zip SampleApp.ipa'

                devicefarm appArtifact: 'SampleApp.ipa', appiumJavaJUnitTest: '', appiumJavaTestNGTest: '', appiumNodeTest: '', appiumPythonTest: '', appiumRubyTest: '', appiumVersionJunit: '1.4.16', appiumVersionPython: '1.4.16', appiumVersionTestng: '1.4.16', calabashFeatures: '', calabashProfile: '', calabashTags: '', deviceLatitude: 47.6204, deviceLocation: false, deviceLongitude: -122.3941, devicePoolName: 'Top Devices', environmentToRun: 'StandardEnvironment', eventCount: '', eventThrottle: '', extraData: false, extraDataArtifact: '', ifAppPerformanceMonitoring: true, ifBluetooth: true, ifGPS: true, ifNfc: true, ifSkipAppResigning: false, ifVideoRecording: true, ifVpce: false, ifWebApp: false, ifWifi: true, ignoreRunError: false, isRunUnmetered: false, jobTimeoutMinutes: 60, junitArtifact: '', junitFilter: '', password: '', projectName: 'SampleApp', radioDetails: false, runName: '${BUILD_TAG}', seed: '', storeResults: false, testSpecName: 'Default TestSpec for Android Appium Java Junit', testToRun: 'BUILTIN_FUZZ', uiautomationArtifact: '', uiautomatorArtifact: '', uiautomatorFilter: '', username: '', vpceServiceName: '', xctestArtifact: '', xctestFilter: '', xctestUiArtifact: '', xctestUiFilter: ''

            }

        }

    }

}

Listing 10-3

Adding Device Farm Stage to Jenkins Pipeline

In the Device Farm stage added now, there are multiple steps defined. First the application is built for testing with xcodebuild to generate a test artifact. From the test artifact generated, an ipa file is created and stored in a location where it is then picked up by the Device Farm plugin for a Device Farm test run.

An environment variable is also used in that stage for TEAM_ID. This environment variable is defined in the environment section of the pipeline definition. The value for the environment variable is dynamically retrieved from AWS Secrets Manager at runtime.

When these changes are committed and pushed to the AWS CodeCommit repository, the Pipeline triggers. A successful execution of the pipeline defined in Listing 10-3 is shown in Figure 10-10.

Figure 10-10

Pipeline execution for AWS Device Farm test

The test results can also be visualized on AWS Device Farm as shown in Figure 10-11.

Figure 10-11

Jenkins pipeline AWS Device Farm test result

Fastlane Build Stage

When all your application tests pass, usually the next step is to build your application. Here, we will add a stage that builds the application for App Store distribution after the Fastlane and Device Farm tests succeed.

Listing 10-4 shows a code snippet for appending a Fastlane build stage to the existing pipeline Jenkinsfile.

pipeline {

    agent any

    environment {

        TEAM_ID = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString ​--output text | jq -r .TEAM_ID'

        ).trim()}"""

        AWS_ACCESS_KEY_ID = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString ​--output text | jq -r .AWS_ACCESS_KEY_ID'

        ).trim()}"""

        AWS_SECRET_ACCESS_KEY = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString ​--output text | jq -r .AWS_SECRET_ACCESS_KEY'

        ).trim()}"""

        MATCH_PASSWORD = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString ​--output text | jq -r .MATCH_PASSWORD'

        ).trim()}"""

        S3_BUCKET = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString ​--output text | jq -r .S3_BUCKET'

        ).trim()}"""

        APP_IDENTIFIER = """${sh(

            returnStdout: true ,

            script: 'aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString ​--output text | jq -r .APP_IDENTIFIER'

        ).trim()}"""

        APPLE_DEVELOPER_USERNAME = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString ​--output text | jq -r .APPLE_DEVELOPER_USERNAME'

        ).trim()}"""

    }

    stages {

        stage("Run fastlane test") {

            steps{

                       .

                       .

            }

        }

        stage("Run AWS Device Farm test") {

            steps {

                        .

                          .

            }

        }

        stage("Build app") {

            steps{

                sh """

                export LC_ALL=en_US.UTF-8 && export LANG=en_US.UTF-8

                fastlane build

                """

            }

        }

    }

}

Listing 10-4

Adding Fastlane build stage to pipeline

In the Fastlane build stage added, the steps include setting some Fastlane-specific environment variables and calling the build lane defined in the Fastfile. Additional environment variables required by Fastlane are also defined. All these environment variable values are dynamically retrieved from AWS Secrets Manager at runtime.

After pushing the changes to the AWS CodeCommit repository, a successful execution of the pipeline defined in Listing 10-4 is shown in Figure 10-12.

Figure 10-12

Pipeline execution for application test and build

Fastlane Deliver Stage

A successful iOS build means the application is signed and ready to be distributed for TestFlight testing or submission to App Store. Here we will add a stage to the pipeline that calls the Fastlane Testflight lane configured in the application to deliver the application to App Store Connect for beta testing with TestFlight.

Listing 10-5 shows code snippet for appending a deliver stage to the existing pipeline Jenkinsfile to deliver the built artifact to TestFlight.

pipeline {

    agent any

    environment {

        TEAM_ID = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString ​--output text | jq -r .TEAM_ID'

        ).trim()}"""

        AWS_ACCESS_KEY_ID = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString ​--output text | jq -r .AWS_ACCESS_KEY_ID'

        ).trim()}"""

        AWS_SECRET_ACCESS_KEY = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString ​--output text | jq -r .AWS_SECRET_ACCESS_KEY'

        ).trim()}"""

        MATCH_PASSWORD = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString ​--output text | jq -r .MATCH_PASSWORD'

        ).trim()}"""

        S3_BUCKET = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString ​--output text | jq -r .S3_BUCKET'

        ).trim()}"""

        APP_IDENTIFIER = """${sh(

            returnStdout: true ,

            script: 'aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString ​--output text | jq -r .APP_IDENTIFIER'

        ).trim()}"""

        APPLE_DEVELOPER_USERNAME = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString ​--output text | jq -r .APPLE_DEVELOPER_USERNAME'

        ).trim()}"""

        APP_STORE_CONNECT_ISSUER_ID = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id app-store-connect-secret --query SecretString --output text | jq -r .APP_STORE_CONNECT_ISSUER_ID'

        ).trim()}"""

        KEY_ID = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id app-store-connect-secret --query SecretString --output text | jq -r .KEY_ID'

        ).trim()}"""

        API_KEY = """${sh(

            returnStdout: true,

            script: 'aws secretsmanager get-secret-value ​--secret-id app-store-connect-secret --query SecretString --output text | jq -r .API_KEY'

        ).trim()}"""

    }

    stages {

        stage("Run fastlane test") {

            steps{

                              .

                              .

            }

        }

        stage("Run AWS Device Farm test") {

            steps {

                              .

                              .

            }

        }

        stage("Build app") {

            steps{

                              .

                              .

            }

        }

        stage("Deliver app") {

            steps{

                sh """

                export LC_ALL=en_US.UTF-8 && export LANG=en_US.UTF-8

                fastlane testflight

                """

            }

        }

    }

}

Listing 10-5

Adding Fastlane deliver stage to pipeline

In the deliver stage added here, the steps include setting up some Fastlane-specific environment variables and calling the Testflight lane defined in the Fastfile. Additional environment variables required by Fastlane are also defined to upload the build artifacts to App Store Connect.

A successful execution of the pipeline defined in Listing 10-5 is shown in Figure 10-13.

Figure 10-13

Pipeline execution for application test, build, and delivery

AWS CodePipeline

AWS CodePipeline is AWS’s managed CICD service. With CodePipeline you can automate the different phases of your software release process without worrying about managing the pipeline infrastructure. It also integrates with different AWS services and third-party tools in different phases to enhance software development, for example, it integrates with Device Farm for application testing, GitHub for source control, and with Jenkins for custom build and test phases.

In the following sections, I will cover how to prepare a Jenkins environment for AWS CodePipeline integration and then dive deep into the integration by demonstrating how to run Fastlane tests and AWS Device Farm tests with a CICD pipeline on AWS CodePipeline. Using the concepts that will be shown, you will be able to expand your CICD pipeline to cover more scenarios.

Setting Up Jenkins Environment for AWS CodePipeline

To configure a Pipeline on AWS CodePipeline that calls Jenkins for custom actions, an external Jenkins server must be in place and that Jenkins server must be configured to enable interaction with AWS CodePipeline service.

Let’s explore some of the prerequisite tasks that must be in place to accomplish our end goal.

Setting Up macOS Build Server

In this section, I will use a Jenkins controller and agent architecture. Refer to Chapter 4 where I covered how to provision an EC2 macOS server and connect to it, and refer to Chapter 5 where you saw how to set up a Jenkins controller on Linux and iOS build agent architecture with EC2 macOS.

When set up successfully, an online macOS build server will look like Figure 10-14.

Figure 10-14

EC2 macOS build server online

Verify Fastlane is installed on the macOS build agent as shown in Listing 10-6.

AWS CLI will be used to retrieve all Fastlane secrets stored in AWS Secrets Manager during Jenkins build so AWS CLI installation and configuration is required on the macOS server.

AWS CLI can be verified and configured as shown in Listing 10-7.

$ aws –-version

$ aws configure

Listing 10-7

Verifying and configuring AWS CLI

jq is also used for parsing jSON data during builds hence it should be installed in the macOS server. It can be installed with Homebrew, as shown in Listing 10-8.

$ brew install jq

Listing 10-8

Verifying jq installation

Setting Up Xcode and Keychain Access

Xcode must be installed and configured using the system user that will be used for Jenkins jobs.

Often, a new user is created for Jenkins on build agents, for example, in Chapter 5 for setting up Jenkins Controller and build agent, the Jenkins controller will use a Jenkins user named “Jenkins” to connect to the macOS build agent via SSH to submit build jobs.

Xcode must be installed and configured on the EC2 Mac instance using the Jenkins user, not the default ec2-user. Refer to Chapter 4 on how to create a password for the Jenkins user and connect to your EC2 Mac instance UI with the Jenkins user on a VNC.

Furthermore, ensure that app signing is configured correctly in Xcode on the build server. Figure 10-15 shows an example of configuring signing and capabilities on Xcode on the EC2 Mac server for the sample application that will be used for demonstration.

Figure 10-15

Signing & Capabilities settings on Xcode

When running a CICD process on Jenkins with Mac server as a build agent, the build often fails because user interaction is not available to provide keychain password when access to a certificate is required for signing build artifacts.

To prevent this, one of the steps I will perform is configure the signing certificate on the build server keychain such that it allows access without prompting for password when Xcode tries to use it during pipeline execution.

The scenario shown in Figure 10-16 covers allowing access to a certificate in keychain. But before items in keychain can be used, the keychain itself must be unlocked. While this can be automated during pipeline execution, the keychain password is required to unlock it. In Figure 10-17, I am storing my keychain password in AWS Secrets Manager so it can be retrieved during pipeline runtime.

Figure 10-16

Configuring signing certificate permision on keychain

Figure 10-17

Storing keychain password in AWS Secrets Manager

Setting Up EC2 IAM Role

When integrating Jenkins with AWS CodePipeline, the Jenkins server must have permission to interact with the AWS CodePipeline service to poll for jobs, report job status, etc. For permissions required, there is an AWS-managed policy AWSCodePipelineCustomActionAccess that contains all the required permissions. Secrets are also stored in AWS Secrets Manager and are retrieved during pipeline execution hence permission to fetch these secrets must be provided to Jenkins server.

When using EC2 for Jenkins servers, a single Jenkins EC2 IAM role can be used across all Jenkins nodes (Controller and build agents) to satisfy the permission requirements. Figure 10-18 shows an example permission configuration for a Jenkins EC2 IAM role.

Figure 10-18

Permission configuration for Jenkins EC2 IAM role

The IAM role must be attached to all the Jenkins nodes (controller and macOS build agents). Figure 10-19 shows an example of attaching an EC2 IAM role to an existing instance.

The example process shown in Figure 10-19 should be repeated for all the EC2 Jenkins nodes in use.

Figure 10-19

Attaching an IAM Role to Jenkins instances

The IAM role attached can be verified from within the Jenkins instances. Listing 10-9 shows an example of how to verify the AWS identity being used. The IAM role name attached should be returned in the response.

$ aws sts get-caller-identity

{

    "UserId": "AROASOKDM2HTHY4MLKGHY:i-0846273b98d6d42f4",

    "Account": "123456789101",

    "Arn": "arn:aws:sts::123456789101:assumed-role/ec2-mac-role/i-0846273b98d6d42f4"

}

Listing 10-9

Verifying AWS identity used on Jenkins instances

Installing AWS CodePipeline Jenkins Plugin

To complete the Jenkins environment setup, the AWS CodePipeline Jenkins plugin can be installed from the Jenkins plugin center, as shown in Figure 10-20.

Figure 10-20

Installing AWS CodePipeline Jenkins Plugin

Setting Up CodePipeline on AWS Console

A pipeline can be created through various means such as AWS CLI, AWS SDKs, AWS Console, etc. I will be using the AWS Console to set up a pipeline. From the CodePipeline console, pipeline creation starts as shown in Figure 10-21.

Figure 10-21

Creating pipeline from AWS Console

For pipeline settings, provide a name and select service role configuration. In Figure 10-22, I chose to create a new service role for the pipeline.

Figure 10-22

Configuring pipeline settings

Next is to configure the source stage of the pipeline. In Figure 10-23, I configure the CodeCommit repository where the sample application code is stored as the source.

Figure 10-23

Configuring pipeline source stage

When creating a Pipeline on the AWS console, the next stage after source defaults to Build and this cannot be modified. Also, a pipeline must contain at least two stages, so in Figure 10-24, I configure a temporary build stage with no real values to work around the limitation for now.

Figure 10-24

Configuring temporary build stage

With the source and build stage, I don’t need any further stages for now, so I skip the deploy stage as shown in Figure 10-25 and proceed with the pipeline creation.

Figure 10-25

Skipping deploy stage creation

After pipeline creation, it automatically starts. Since the build stage was configured as temporary with no real values, it will fail. You can either let the pipeline fail or stop the pipeline execution. As shown in Figure 10-26, I stopped the pipeline execution.

Figure 10-26

Stopping pipeline execution after creation

Now that the basic pipeline has been set up, in the following, I will show you how to extend this pipeline to integrate with Jenkins and AWS Device Farm for iOS app testing.

Fastlane Testing with AWS CodePipeline and Jenkins

For this integration, you need to set up a CodePipeline-enabled Jenkins project first, then configure a custom stage on CodePipeline to integrate with the Jenkins project.

Configuring Jenkins Project to Test App

Here, I will set up a new Jenkins freestyle project, as shown in Figure 10-27, and configure it to run a Fastlane test.

Figure 10-27

Creating Jenkins freestyle project for Fastlane testing

Take note of the project name, this will be used for configuration on AWS CodePipeline.

Since I am using a controller and build agent architecture for my Jenkins environment, I will delegate this build to my iOS build agent using labels as shown in Figure 10-28.

Figure 10-28

Setting where the project will be run

We need to configure AWS CodePipeline for source code management since the source code for this project will be passed by AWS CodePipeline. My source code management settings are shown in Figure 10-29.

Figure 10-29

Configuring test project source code management

The values provided for Provider in the CodePipeline Action Type configuration should be noted. This will be used in the CodePipeline configuration.

The build trigger invokes the build. To ensure that the build is invoked when appropriate, I will configure the project to continuously poll CodePipeline to check if there are any pending jobs that matches its configuration. As shown in Figure 10-30, my project will poll CodePipeline every minute for any pending jobs.

Figure 10-30

Configuring build trigger for project

The build script for Fastlane testing is shown in Listing 10-10.

set +x

echo "Set fastlane variables..."

export LC_ALL=en_US.UTF-8

export LANG=en_US.UTF-8

echo "Starting Fastlane..."

fastlane test

Listing 10-10

Fastlane testing build script

The script should be entered into the build command section of the project as shown in Figure 10-31.

Figure 10-31

Entering build script for project

The last step in the project configuration is to add an AWS CodePipeline Publisher post build action, as shown in Figure 10-32. Provide an artifact name to identify any artifacts published from the Jenkins project.

Figure 10-32

Configuring AWS CodePipeline publisher post build action

Take note of the artifact name configured as it will be used in CodePipeline configuration.

Adding Custom Stage to CodePipeline

To add the Jenkins project created to AWS CodePipeline, edit the pipeline created as shown in Figure 10-26. Delete the previously configured temporary build stage and add a new stage as shown in Figure 10-33.

Figure 10-33

Deleting existing build stage and adding new stage to pipeline

In the newly added stage, add a new Jenkins test action as shown in Figure 10-34.

Figure 10-34

Adding new Jenkins test action to the new stage

The Add Jenkins action must be selected under the Test category to match with the configuration in Jenkins project.

Configure the Jenkins test action. The Provider name, Project name, and Output artifacts field can be retrieved from the values provided and noted when configuring your Jenkins project. In the Server URL field, enter the public IP address of your Jenkins controller. Figure 10-35 shows the configuration of my Jenkins test action.

Figure 10-35

Configuring Jenkins test action

Once the action is configured, you will now have a two-stage pipeline. To start pipeline execution, you can either push new commits to the CodeCommit repository or manually release change on the CodePipeline console.

Figure 10-36 shows a successful pipeline execution for Fastlane testing.

Figure 10-36

Successful pipeline execution for Fastlane testing

For more information on Jenkins project execution, build logs can be retrieved from Jenkins. Listing 10-11 shows a snippet of the logs generated on Jenkins for the pipeline run shown in Figure 10-36.

Listing 10-11. Log snippet for Fastlane testing CodePipeline execution

AWS Device Farm Integration

It is imperative to add testing on real iOS devices to your development process, I will demonstrate how to add a stage for AWS Device Farm testing to the existing CICD Pipeline. Similar to the previously configured Fastlane stage, I will first set up a CodePipeline-enabled Jenkins project for building the app, after which I will set up an AWS Device Farm Project that will be used for testing, then finally configure a Device Farm stage on CodePipeline with separate actions integrating with Jenkins and AWS Device Farm.

Configuring Jenkins Project to Build App for Testing

To test an iOS app on Device Farm, the ipa package is needed. The project created here will be used to generate the testable ipa package for Device Farm use.

This requires a new Jenkins freestyle project to be created, as shown in Figure 10-27. I will also ensure this project runs on my iOS build agent, as shown in Figure 10-28. For Source Control Management, while it is like the previous project that was created, I will change some fields to suit this current use case, as shown in Figure 10-37.

Figure 10-37

Configuring build project source control management

In Figure 10-37, the CodePipeline Action Type category is set to Build because this project is configured for a build action, and it will be configured as such on CodePipeline.

Take note of the Provider and Jenkins project name as this will be used in CodePipeline configuration.

The build trigger for this project will be configured like the previous project created for Fastlane testing, as shown in Figure 10-30.

The build script that will be used for this project is shown in Listing 10-12.

set +x

export TEAM_ID=`aws secretsmanager get-secret-value --secret-id fastlane-secrets --query SecretString --output text | jq -r .TEAM_ID`

export KEYCHAIN_PASSWORD=`aws secretsmanager get-secret-value ​--secret-id fastlane-secrets --query SecretString --output text | jq -r .KEYCHAIN_PASSWORD`

security -v unlock-keychain -p "$KEYCHAIN_PASSWORD" "$HOME/Library/Keychains/login.keychain"

/usr/bin/xcodebuild build-for-testing -scheme SampleApp ​-destination generic/platform=iOS DEVELOPMENT_TEAM=$TEAM_ID ​-allowProvisioningUpdates -derivedDataPath $WORKSPACE

mkdir Payload && cp -r $WORKSPACE/Build/Products/Debug-iphoneos/SampleApp.app Payload/

zip -r Payload.zip Payload && mv Payload.zip SampleApp.ipa

Listing 10-12

App build for test script

As shown in the build script, the team ID and keychain password are retrieved from AWS Secrets Manager. The keychain password is used to unlock the keychain prior to building the app for testing with xcodebuild.

The script should be entered into the build command section of the project, as shown in Figure 10-38.

Figure 10-38

Enter build script for project

The last step in the project configuration is to add an AWS CodePipeline Publisher post build action, as shown in Figure 10-39. Provide an artifact name to identify any artifacts published from this Jenkins project.

Figure 10-39

Configuring AWS CodePipeline publisher post build action

Take note of the artifact name configured as it will be used in CodePipeline configuration.

Setting Up AWS Device Farm

I will be using the Device Farm Project that was created in Chapter 9. The project that will be used is shown in Figure 10-40.

Figure 10-40

AWS Device Farm project

To submit test runs, you also need to have a device pool. In this example, I will be using the managed “Top Devices” device pool to test my app. To use this device pool from AWS CodePipeline, I must provide the Device Pool ARN, so to get the ARN, using the project ARN that can be retrieved from Figure 10-40, I will use the AWS CLI as shown in Listing 10-13.

$ aws devicefarm list-device-pools --arn arn:aws:devicefarm:us-west-2:123456789101:project:3973dab9-33ab-4d34-87c6-3e183e009b07 --region us-west-2 --query 'devicePools[?name==`Top Devices`]'

Listing 10-13

Getting device pool ARN

Take note of the Device Farm Project ID shown in Figure 10-40 and the Device pool ARN returned by running command in Listing 10-13 as these will be used in the pipeline configuration.

Adding Device Farm Stage to Pipeline

I have configured a Jenkins project that can build my app for testing and generate artifacts and I have a Device farm project ready for use. Now I will add a new stage to the Pipeline.

Edit the pipeline and add a new stage to make it a three-stage pipeline. Figure 10-41 shows the name I gave to the new stage.

Figure 10-41

Adding a Device Farm stage

After adding the new stage, you will configure the first action. This will be a Jenkins build action that will connect to the Jenkins project that was created earlier. Figure 10-42 shows how to add a Jenkins build action.

Figure 10-42

Adding a Jenkins build action

The Add Jenkins action must be selected under the Build category to match with the configuration in the Jenkins project.

The Provider name, Project name, and Output artifacts field can be retrieved from values provided and noted when configuring your Jenkins project. In the Server URL field, enter the public IP address of your Jenkins controller. Figure 10-43 shows the configuration of my Jenkins build action.

Figure 10-43

Configuring Jenkins build action

Next, within the same stage, we will configure the Device Farm action. The Device Farm action cannot run in parallel with the Jenkins action because Device Farm depends on the artifacts generated from the Jenkins action. A new action group can be added as shown in Figure 10-44.

Figure 10-44

Adding Device Farm action

The Device farm action can be configured as shown in Figure 10-45. AWS Device Farm should be selected as the Test provider. The output artifact configured in the Jenkins action must be configured as the input artifact in this action so artifacts can be exchanged. The project ID and Device Pool ARN retrieved from Device farm should be provided here. AppType, path to the ipa file within the input artifact, and TestType are also required fields.

Figure 10-45

Configuring Device Farm action

AWS Device Farm is only available in US-West-2 region (Oregon), so this must be the region selected in the Pipeline action.

After configuring the two actions, you should have a stage that looks like Figure 10-46.

Figure 10-46

Device Farm stage configured with two actions

Once the action is configured, you will now have a three-stage pipeline. To start pipeline execution, you can either push new commits to the CodeCommit repository or manually release change on the CodePipeline console.

Figure 10-47 shows a successful pipeline execution for Fastlane testing and AWS Device Farm test.

Figure 10-47

Pipeline execution for Fastlane testing and Device Farm test

The build logs for the application build can be retrieved from Jenkins, while details of the Device Farm test result can be seen from the Device Farm console.

Summary

A CICD pipeline enables you to orchestrate different steps and components of your application development starting from source control to deployment to users. In this chapter, I demonstrated how different flavors of CICD pipelines such as Jenkins Pipeline and AWS CodePipeline can be used to orchestrate the different concepts we have covered in previous chapters using real examples.

In Jenkins Pipeline, we saw how an application can be tested, built, and delivered to App Store Connect. In AWS CodePipeline, we covered different testing options while also integrating with Jenkins. AWS CodePipeline can be further expanded to add additional stages like the Jenkins Pipeline. For example, after a Device Farm test you can add an additional stage that integrates with Jenkins to build an application for distribution with Fastlane and even deliver the application to App Store Connect.

Index

A

Amazon CloudWatch

Amazon EC2 Mac

AWS CLI

AWS CloudFormation

AWS Management Console

cleaning up

CloudWatch

connecting SSH

Core i7 processors

limits/pricing

terraform

VNC client

Amazon Elastic Compute Cloud (Amazon EC2)

Amazon Elastic Container Service (Amazon ECS)

Amazon Machine Image (AMI)

Amazon Resource Name (ARN)

Amazon Web Services (AWS)

AWS CodeCommit

CodeArtifact

CodeBuild

continuous delivery

continuous integration

definition

IaC

CDK

CloudFormation

monitoring/logging

Amazon CloudWatch

AWS CloudTrail

CloudWatch alarms

CloudWatch logs

CloudWatch metrics

definition

appspec.yaml

App Store Connect

API key

base64 encoded

creation

dashboard

downloading

Keys tab

name/access level

request access

App Store lane to Fastfile

app submission

AWS Secrets Manager

creating new app

definition

Fastlane

identifiers

Jenkins project

configuration

triggering

portal

sceenshots

TestFlight

TestFlight lane to Fastfile

Xcode, uploading build

AWS CodePipeline

AWS Console

build stage

configuration

creating pipeline

deploy stage

pipeline execution

source stage

AWS Device Farm

action group

actions

adding stages

artifact name

build logs

build script

build trigger

configuration

ipa package

Jenkins build action

Jenkins freestyle project

pipeline execution

Publisher post build action

setting

three-stage pipeline

US-West-2 region

Fastlane testing

build script

build triggers

deletion/addition, stages

Jenkins freestyle project

Jenkins test action

log snippet

pipeline execution

Publisher post build action

setting

source code management

Jenkins environment

EC2 IAM role

Jenkins plugin

Jenkins server

macOS build server

Xcode/keychain access

AWS Device Farm

automated testing

AWS CLI, test runs

application archive

ARN

arn project

AWS Console

creation

device pool/output

scheduled run

scheduling

status

test report

uploading

upload status

AWS console, test runs

accessing

Built-in Fuzz test

SeeBuilt-in Fuzz test

built-in test types

creation

testing frameworks

XCTest UI test

SeeXCTest UI test

components

definition

ipa package

application archive

creation

directory location

Payload directory

testing

Jenkins plugin

configuration

configuring permission

credentials

IAM credentials

IAM policy

IAM role

IAM user

installation

Jenkins project

build script

configuration

post build actions

running tests

SampleApp.ipa

test environments

test framework/execution environment

triggering

terminologies

testing

B

Built-in Fuzz test

application ipa archive

automated test run

device pool

device state

pending state

run details

run name

run status

selection

test report

C

Cloud Development Kit (CDK)

CodeArtifact

CodeCommit

branching

AWS CLI

AWS console

default branch

Git

local repository

CLI

CloudFormation

creation

definition

resources

stack details page

connection URLs

Git repository

cloning

deletion

migration

pushing contents

GRC

SeeGit remote codecommit (GRC)

HTTPS

SeeHTTPS

pull requests

SeePull requests

pushing code

Python Boto3 SDK

creation

credentials

documentation

installation

Python script

SSH

SeeSSH

Terraform

definition

initialization

script

terraform plan

CodeDeploy

CodePipeline

Command line interface (CLI)

ContentView()

Continuous Integration Continuous Delivery (CICD)

D

DevOps

E

Elastic Block Storage (EBS)

F

Fastlane

adding lane

Amazon S3

AWS Secrets Manager

certificates/keys

code signing concepts

definition

environment variables

features

initialize match

installation

Jenkins project

configuration

triggering

Matchfile

match password

match secrets/parameters

output

S3 bucket

uses

Xcode

G

Git

branch

definition

fundamentals

installation

pull requests

repository

changes/status

cloning/working

initialization

GitLab

features

project

blank project

configuration

creation

runner

CICD settings

download

installation

macOS

registration

token/URL

Git remote codecommit (GRC)

connection

credentials

definition

IAM role

installation

temporary credentials

workstation

H

HashiCorp Configuration Language (HCL)

HTTPS

creation

Git credentials

Security credentials, IAM

uses

I

Identity access management (IAM)

Infrastructure as code (IaC)

Integrated development environment (IDE)

iOS application development

Apple developer account

command line tool

Mac

Swift

J, K, L

Jenkins

agents

controller

Amazon Linux AMI

EC2 instance

initial admin user

initial setup page

installation

instance/storage capacity

package manager

password

plugins

rpm key

security group configuration

service

software packages

status

welcome page

yum repo

EC2 Mac

adding node

build nodes

console output

creating user

credentials configuration

environment variables

host key verification

installation

label

options

security group configuration

SSH credentials

SSH keys

symbolic link

Jenkins AWS CodeCommit plugin

access

commands

configure

console

CodeCommit trigger

SNS

SQS

SQS polling/viewing message

SQS publish message

SQS queue access

SQS receive message

create

HTTPS

install

manage

SCM

search

Terraform

testing

Jenkins environment

AWS CLI

Fastlane

JSON query language (jq)

Xcode

Jenkins Pipeline

build triggers

configuration

creation

definition

Device Farm stage

accurate syntax

environment variable

ipa file

Jenkinsfile

pipeline execution

Pipeline script

Pipeline syntax generator

selection

test artifact

test results

Fastlane build stage

environment variables

Jenkinsfile

pipeline execution

Fastlane deliver stage

Fastlane test stage

errors

execution logs

single-stage Pipeline

Jenkinsfile

repository structure

script path

spaces

syntaxes

M, N, O

Mac

macOS instance

APFS container

disk/sizes

reboot

storage

volume modification

volume size

P, Q, R

Pull requests

approval rules template

approve option

AWS CLI

AWS console

creation

merging

CodeGuru reviewer

association

definition

reviewing

creation

AWS CLI

AWS console

test branch

merging

AWS CLI

AWS console

S

Secure shell (SSH) protocol

config file permission

configuration

connection

key pair

public key

adding

key ID

retrieving

uploading

Simple Notification Service (SNS)

Simple Queue Service (SQS)

Source Code Management (SCM)

Swift

building application

iOS simulator

physical iPhone

creating application

app icon

developing Swift

template

Xcode live preview

definition

installing Xcode 12

Xcode

T

Terraform

TestFlight

U

User interface (UI)

V, W

Virtual network computing (VNC) protocol port

Virtual Private Cloud (VPC)

X, Y, Z

Xcode

command line tools

iOS SDK License Agreement

welcome page

XCTest UI test

application/test package

app package

custom device pool

running, test

test bundle