AI for Creative Production

Copyright © 2025 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Early Access Publication: AI for Creative Production

Early Access Production Reference: B34247

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK

ISBN: 978-1-80602-581-7

www.packt.com

AI for Creative Production: A handbook for the ethical use of AI in creating and processing text, images, video, and audio

Welcome to Packt Early Access. We're giving you an exclusive preview of this book before it goes on sale. It can take many months to write a book, but our authors have cutting-edge information to share with you today. Early Access gives you an insight into the latest developments by making chapter drafts available. The chapters may be a little rough around the edges right now, but our authors will update them over time.You can dip in and out of this book or follow along from start to finish; Early Access is designed to be flexible. We hope you enjoy getting to know more about the process of writing a Packt book.

Chapter 1: The AI Story So Far

Chapter 2: Ethical Implications of AI Use

Chapter 3: Utility AI with Audio

Chapter 4: Utility AI with Images and Videos

Chapter 5: Utility AI with Text

Chapter 6: Generative AI with Text

Chapter 7: Generative AI with Images

Chapter 8: Generative AI with Video

Chapter 9: Generative AI with Audio

Chapter 10: Automation AI with Images

Chapter 11: Automation AI with Video

Chapter 1: The AI Story So Far

Join our book community on Discord

https://packt.link/EarlyAccess

Artificial Intelligence (AI) has captured our collective imaginations since Alan Turing first asked whether machines could think. While we still can't give a firm answer to that big question, the increasing pace of change in the AI world means that creative white collar workers are, for the first time, fearing for their careers.Computers themselves have transitioned from toys for enthusiasts, to tools for professionals, to enablers for everyone—and AI is on the same trajectory, over a much shorter time span. In the same way that smartphones have made computing easier for regular people, AI promises to make creativity itself easier for a wider audience. Many in the creative space have now felt, or started to feel, the impacts of AI on their professional lives.However, the imperfect nature of AI tools means that there's still a huge role for humans in creative production. Even though its help is imperfect, AI will still help us to find video clips more easily, fix bad audio more quickly, check our ideas against virtual collaborators, and, in many ways, be able to reach just a little bit further out of our comfort zones.Disruption is certain, but AI will not end human creativity. This book aims to show creative people how they can harness the power of many different AI-based tools to make their creations better.In this chapter, we'll cover the following main topics:

The early days of AI

How AI works today

Uses of AI

The shortcomings of AI

Apps, tools, and technologies

To begin, let's look back.

The early days of AI

Neural networks—connected virtual nodes that work in a similar way to human brains—were first proposed in 1944, and they've experienced varying success in the decades since. A neural network is first trained by presenting a set of inputs and the matching outputs it should learn to produce. During the training process, each node in the interconnected network becomes stronger or weaker, assigned weights that reinforce some connections and de-emphasize others. Essentially, it's learning to recognize patterns without ever being taught a rule explicitly.After the training, a neural network can be fed a new input to produce a new, hopefully correct output. However, this indirect approach is quite different from other rule-based (or heuristic) methods that other AI systems have used, and many years passed before computing power unleashed its potential.Probably the best-known event in AI history occurred when Alan Turing presented an imitation game, a test in which an interrogator exchanges messages with a human and a digital participant, then tries to identify which is which. This became known as the Turing test, and for many decades, it was seen as an insurmountable bar.In 1965, a program called ELIZA simulated a psychotherapist, repeating keywords from a user's input in its own responses. Techniques such as this became foundational in natural language processing, and although this technique is simple, it was enough to fool many people into thinking they were talking to a human.Soon after, in the 1970s and 1980s, another focus of early AI was expert systems, which aimed to capture knowledge in a specific niche in order to answer related questions. Using heuristic methods rather than neural networks, it proved too difficult to codify the knowledge and deal with edge cases.AI research continued, but the breathless hype was not met with reality, and enthusiasm waned. Computers continued to get faster and user interfaces became more friendly, but AI progress stalled. In the mid-1990s, I majored in artificial intelligence and artificial life in an information technology degree, implementing my own version of ELIZA, training neural networks, and simulating virtual plants that I then evolved against one another. While it was fascinating to see how these techniques could be used, there was a huge gap between student projects and practical applications.Heuristic approaches and neural networks found specific uses in limited areas, and as computers became more powerful, more uses became possible. Fuzzy logic allowed computers to read messy human handwriting, and rules governing enemy behavior governed computer-controlled enemies in video games, in real time. Language translation, though imperfect, became a common feature in many applications.However, it was Google's introduction of transformer models in 2017 (in a paper entitled Attention Is All You Need) that led to the modern explosion in the power and utility of AI, alongside a breakthrough in diffusion models for image generation. Let's take a high-level look at how these modern systems work, while noting that we don't truly understand all the details.

How AI works today

Transformer models include a mechanism called attention, which identifies the most important parts of an input and assigns them a higher level of importance. Think of the adjectives in this sentence being assigned less importance than the nouns or the verb:The brown dog attacked the timid cat aggressively.(While I won't be digging into the math here, the paper Attention Is All You Need is linked in the Further reading section of this chapter.)The transformer model was intended for use with translation, but its true potential became clear when it was combined with Large Language Models (LLMs). An LLM is a predictive model trained to produce output a word at a time, based on the words that had come before—like an advanced version of autocorrect. A sentence such as "The ice cream was…" is more likely to conclude with the adjective "…sweet" or "…delicious" than, for example, "dusty."Adding transformers to LLMs with large pools of training data created—or seemed to create—emergent abilities, such as the ability to summarize. These models can, at some level, perform reasoning tasks, and it's not yet fully clear exactly how they do it.Although we did train these models on vast amounts of text, we did not train models to summarize or reason, and yet, they can do that. It's not clear if there's any deep understanding going on, but does it matter? If you ask for a summary of an article and you get one that's accurate, that's good enough to be useful. Transformer-backed LLM systems are widely used today for help, advice, information, and even—though they aren't sentient—companionship. Modern LLMs include OpenAI's ChatGPT, Anthropic's Claude, Meta's Llama, and Google's Gemini, to name a few.The other side of the modern AI story belongs to image generation. While ChatGPT's image generator uses an autoregressive transformer model, diffusion models are more common, starting with DALL·E and continuing with Midjourney, Stable Diffusion, and others. A diffusion model starts with random noise, then repeatedly applies a neural network to predict how noise should be removed until it approximates the requested text prompt.Training is a huge part of this process, involving a forward diffusion process of adding noise to these images, and a reverse diffusion process where the noise is removed. By feeding these models immense numbers of images, we have been able to improve their outputs to the point where they can, at least some of the time, create synthetic images that look real. Videos can be similarly synthesized using a more complex process, but Machine Learning (ML)—the general term used for the process by which a computer can be trained to perform tasks—can be used for many different tasks.So, what are those tasks?

Modern applications of AI

The term AI has been overused, and for many people, AI simply now means "a computer made it"—but there's so much more to AI than that. For clarity, this book will separate ML and AI applications into three broad categories. They are the following:

Utility AI: Recognition, classification, and understanding models, which can be used to help sort and manage the data they are presented with

Generative AI: Creation of text, images, video, audio, and even programming code

Automation AI: Directed control of other computer systems to perform tasks that humans usually would

Let's dig deeper. If you're asking questions of a chatbot, that's likely utility AI. If you're creating music or original text, that's generative AI (or GenAI for short). And if you're using Siri or Google Assistant to create a calendar appointment for you, that's a simpler form of automation AI. The names of these terms will change over time; the term AI agents (or agentic AI) is a part of automation AI that's becoming more prominent.Each of these areas will be covered in more detail in this book, and there's a part of this book devoted to each of these three sections. While there is certainly crossover between these categories—for example, translation involves recognition of what was said as well as generation in another language—in general, most AI applications today can cleanly fit into one of them.For now, let's treat these categories as broad, basic headings before we drill into the details.Today, just some of the things you might use AI tools to help you with are the following:

Generating ideas for an essay

Assessing written text for completeness or accuracy

Creating photographic-style images

Replacing part of an image with something synthetic

Processing audio to remove noise

Creating synthetic recordings to match a particular person's voice

Writing scripts to automate features in graphic design applications

Asking for help with learning a new application

Creating 3D models based on images or text descriptions

These tasks are just the tip of the iceberg, and I've long held the view that progress will really explode when agents start driving our computers for us. At first, AI models will start using our applications as we watch, but eventually they'll just perform tasks directly, or use lower-level systems that we cannot. Today, we're only at the start of the journey, but things are moving quickly.

Where AI falls short

Few technologies break through to the mainstream, but AI certainly has, and the excitement and fear across creative communities is palpable. However, one factor that's known but frequently ignored is that the output from AI systems is rarely perfect. To the layperson, an AI-generated image can look great, but to a Photoshop expert, there may be obvious flaws. AI-made code will frequently contain references to programming interfaces that simply don't exist. Horrifyingly, AI-generated legal arguments refer to fictional legal cases. Advice is not always accurate, and the more you know about a subject, the more likely you are to find fault with an AI's output.While the quantity of work produced by AI can sometimes make up for the overall lack of quality, in general, most AI output lacks something, tangible or not. This could be for many reasons, but chief among them is likely that AI excels at remixing, rather than original thought. Every image a diffusion model creates is, in some way, derived from the training data it was fed.Although it's certainly been argued (by Kirby Ferguson) that "Everything is a Remix," the best work contains an original spark, and the process of creation often inspires new creations itself.It's critical to realize that the creative process is an important part of any product or artwork. If you skip the brainstorming, planning, and experimentation phases, the finished product will be lacking in ways both obvious and subtle, and you won't have grown as an artist.You might have what you asked for, but creativity can lead to better outcomes than the most obvious ones. Skipping straight to an end goal and omitting all the creative decisions made along the way will produce worse products, because the best designers don't just create exactly what the client asks for.How good can AI be? From everything I've seen to date, a good GenAI produces work that's OK but not great. It's rarely perfect, and it's unlikely to be top-class A+ work. At best, it's a solid B+. But that doesn't mean you shouldn't use AI at all.Instead, use AI for what it's really good at:

Replacing parts of an image, not the whole thing

Extending a photograph's edges to adapt to a size change, not creating whole images from scratch

Picking the best photographs from a full-day shoot, with a human checking its work, not just trusting it completely

Identifying which parts of a music track can be repeated to suit a different duration, not simply creating a whole new piece

Producing temporary art to help with planning, not finished art

Helping you write programming scripts to make creative tasks more efficient, not doing everything by hand

In my opinion, the best uses of AI are to augment human creativity, not trying to replace it entirely. If an AI can help you extend your creative reach a little further, great. But if you skip all the traditional creative steps and create original work that's well outside your comfort zone, you simply won't recognize the mistakes it makes. AI can help you run, but you have to learn to walk first.As well as the risk of unnoticed mistakes, AI also brings the risk of creative stagnation. Though a client may only care about an end product, the process of making it helps an artist to learn. If you skip straight to the end, you learn nothing, and earn no sense of accomplishment. The more you rely on AI, the less likely you are to gain the insights that turn a novice into an expert.With all that in mind, there's still plenty for AI to do: boring, technical, or rote tasks that don't teach you much or simply take too long to be practical, or that computers are simply better at than humans. Explore with your eyes open, and you'll see where AI has value. As a tool, AI can be a great assist, but as a total replacement for an artist, it's lacking.To conclude this chapter, we'll next take a quick look at the different kinds of AI-based services available today.

Apps, tools, and technologies

Change is inevitable, but rarely is the pace of change as fast as it is in this space today. While the tools I've mentioned here are available now, you can expect great new tools to pop up, and some listed here to disappear. This isn't intended to be an exclusive listing of all the tools out there, but it should serve as an introduction to at least some services available today.For many people, I suspect "AI" starts and finishes with chatbots, and likely the best-known one, ChatGPT. While it's true that the various chat-focused LLMs do offer other services, you'll also discover many specialized services with or without a chat-based interface, AI services in apps you may already be using, and a whole world of open source models you may be able to run on your own hardware without a subscription.Let's begin with a look at the best-known LLMs.

LLM-based hosted AI services

If you'd like to handle a multi-faceted problem, using a service that can interpret text, create at least rudimentary images, search the web, analyze documents, and more is what you need. Starting with a service from one of the large, well-known AI companies is probably a good idea. All of these companies allow you some number of queries for free, but if you need more, subscription plans are available. Here, I'll be mentioning the price of the top-level consumer-facing plans rather than the costs of tokens that a developer might face.Some of these models (such as Claude) are strictly text-based, while others (such as ChatGPT) are multimodal, able to analyze images or the spoken word in addition to text. Some can create images, video, or audio, while some cannot. And that's OK, because sometimes you simply want information to help you do your job better, and a conversational interface makes that information much more accessible.While text generation or refinement is an obvious use of an LLM, another key use in the creative space is for help with ideation, learning, or research. While these systems may not be able to create your work for you, they definitely have a place.Each of these services is trained on a unique set of training data, and their models are guided by a unique system prompt—a set of instructions fed to the model before a user's input. As a result, each LLM will have a different feel or personality. Try several, map your needs to their capabilities, and if you choose to subscribe to one or more, re-evaluate your decision frequently. Services in this space are upgraded frequently, but almost all are free to try. Be sure that a service can do what you expect before committing to a subscription, and pay monthly if you do—everything changes quickly!It's also worth noting that many of these services are targeted separately toward regular consumers and developers, who usually access these services at a lower level through a more complex token-based system. Since most creatives aren't developers, we'll assume you're going through the consumer-friendly front door and focus on the fixed monthly prices.This is absolutely not an exhaustive list, but it's unlikely you've heard of every service here. Still, one AI company you'll have almost certainly heard of is…

OpenAI

Just as "googling" has become synonymous with search, "ChatGPT" is often used when talking about any AI chatbot. ChatGPT is OpenAI's flagship product, and it's probably the best-known LLM in the world. You can access ChatGPT through the web, through their apps on many platforms, and through integrations with other products. Several models of ChatGPT have been developed, and the offerings are constantly changing, but it's not limited to just text—you can upload documents and receive updated documents in return. With all of these services, if you're not sure about a model's capabilities, just ask the LLM itself.As well as ChatGPT, OpenAI was once well known for DALL·E, an early image generation tool that's now part of ChatGPT and can be tried for free. Sora is a newer image and video generation tool, but it's only available on the paid plans. At the time of writing, the expensive ChatGPT Pro is required for 1080p video generation, while lower-quality 720p videos can be generated with ChatGPT Plus. Still, it's potentially very useful for creatives, and here's what you can expect to pay:

ChatGPT: Free to start, accessed at https://chatgpt.com

ChatGPT Plus: US$20/month

ChatGPT Pro: US$200/month

Google

Search giant Google offers a number of products under the name Gemini—currently the AI Pro and AI Ultra plans, featuring Gemini Pro and Gemini Nano models alongside other offerings, such as Veo 3. Gemini is an LLM with a wide range of capabilities that's available on the web, in an app, and in Google Workspace, as well as being built directly into Google Search with the newer AI Mode. If you're on Android, it's also possible to use Gemini in place of Google Assistant, but at the time of writing, it cannot replace Apple's Siri on iPhone. It's free to start, but the most advanced capabilities require a Pro plan. There are definite uses in the creative fields, and the pricing is as follows:

Gemini: Start for free, accessed at https://gemini.google.com/app

Gemini AI Pro: US$20/month

Gemini AI Ultra: US$249/month

Meta

Better known as the creator of Facebook and owner of Instagram, Meta has created Llama, their flagship LLM, which has grown into the largest free, nearly open source option. With LLMs, being open source doesn't necessarily mean you can download it and run it locally, because the full-size versions are simply too big, but that doesn't mean it's impossible—more on that in the Running your own AI services section later in the chapter. And being free doesn't make Llama immune from copyright issues either; Meta has faced legal action, accused of training on copyrighted materials, though as long as you don't ask Llama to reproduce copyrighted work, you're unlikely to face issues yourself. There's no need to log in before talking to the LLM, though image and video generation does require that you make a free account. Llama can be accessed at https://meta.ai.

Anthropic

Claude is an LLM available from Anthropic, and like the previously mentioned LLMs, it's happy to chat, analyze images, work with code, and more. However, Claude cannot generate images, so it won't be suitable if you're interested in GenAI for images or video. Pricing is in line with other services:

Claude: Start for free, accessed at https://claude.ai

Claude Pro: US$20/month

Claude Max: US$100-200/month

Microsoft

Copilot is Microsoft's multi-purpose AI tool, and although it can be accessed on the web, it's most commonly found inside Microsoft Office apps. In Word, it acts as a grammar checker and text refinement engine, while in PowerPoint, it can help you with layout, image generation, or creating entire presentations. (It can also help in Excel, but since that's less likely to be useful in a creative context, we won't cover it here.) While it's historically been based on ChatGPT, Microsoft has announced a plan to use in-house models. An account is required, and there are limits on the free plan, but there's currently no "expensive" plan:

Copilot: Start for free, accessed at https://copilot.microsoft.com

Copilot Pro: Fewer limits and more capabilities, US$20/month

DeepSeek

DeepSeek is a free, open source LLM built with a focus on efficiency, and with a few different models available. While the base R1 model only handles text, the Janus model is multimodal and can interpret and produce images. Note that the version of DeepSeek hosted in its native China censors some topics. Reportedly, the downloadable version of DeepSeek does not have these limitations. Here are the options at the time of writing:

DeepSeek: Start for free, accessed at https://deepseek.com (note that registration with an email address from a global email provider is required)

API access and more advanced models require lower-level access

While most well-known AI platforms start with an LLM and have built a variety of services into those chatbots, you don't need to interact with a chatbot to make use of AI services. Some services are more focused.

Special-purpose hosted AI services

For some jobs, you may find it more reliable and predictable to visit a web-based service tailor-made for a specific task. Here are a few image, video, and audio generation services that we'll be looking at later in the book:

Midjourney: This provides high-quality image generation. While the quality is often higher than most other AI image generation providers, the costs are too. With no free trial, users will need to sign up for a paid plan. Though prices start at US$10/month, only the Pro (US$60/month) and Mega (US$120/month) plans allow you to keep your images private.It can be accessed at Midjourney.com.

FLUX: This is one of many image generators you may not have heard of, but worth paying attention to. From Black Forest Labs, this tool can extract items from existing photographs, zoom in and "enhance" like in procedural TV shows, change art style, and make many changes with high quality. It is currently 8c per image for the best model, but with an open source version coming soon.It can be accessed at https://bfl.ai.

Stability.ai: The creators of Stable Diffusion allow you to download their open source generative models for free, but also offer online plans from US$9 to US$99/month. Though they're best known for image generation, their latest models can also generate video, audio, and 3D models. It can be accessed at Stability.ai.

Canva: The well-known online design service has branched into AI, integrating a chatbot that allows you to generate images, change images, and create new designs. While a starting point is undeniably useful for simple design tasks, I suspect most design professionals will need more creative control than Canva's online design tools offer.

Canva is free to start, but Canva Pro costs US$13/month to raise limits on AI generation and grant access to more advanced tools. Note that to generate on-brand text and visual assets, you must use Canva Teams (at a cheaper cost per person for 3+ seats). Interestingly, if you sign up for Canva Enterprise (100+ seats), you can be indemnified against intellectual property claims from AI generation (https://www.canva.com/policies/ESA/). It can be accessed at Canva.com.

Runway: This video-focused AI company offers generative image and video services in a variety of plans ranging from US$0 to US$76/month. Content created with free plans is watermarked, though. For best results with the most advanced models, upload an image to use as the basis of video generation. (If you ask for a video in text form, it'll generate an image first, then animate it.) Many video-related services are included. It can be accessed at https://runwayml.com.

Gling: This online service for video creators helps you start the editing process by automatically removing bad takes and cutting silences and filler words (um, ah), and then using text-based editing to cut your video. It is available from US$0 to US$20/month. It can be accessed at https://gling.ai.

Envato: Though Envato Elements is mostly known as a stock library, they've integrated image, video, music, and voice generation, as well as image editing, into their service. It is available from US$16.50/month, including Google's Veo 3 model for video generation.It can be accessed at https://envato.com.

Descript: This cloud-based video editing platform has expanded its AI offerings recently. Automatic transcription enables text-based editing, and AI avatars can speak with AI-generated voices reading an AI-generated script with AI-generated b-roll images. It can be accessed at https://descript.com.

ElevenLabs: Text-to-speech in synthetic voices and voices cloned from specific real voices. The output can sound just like a real human, with emotion and a varied cadence, and it's even possible to translate spoken audio to another language with the same voice as the original speaker. Audio cleanup tools are also included in the comprehensive offerings.

Cloud-based tools aren't the only way to attack specific tasks, though—existing desktop apps have adapted to the new AI paradigm in a variety of ways, with and without cloud assistance.

AI tools built into existing apps and platforms

Third-party cloud services face competition from the tools that many designers are already using. In many cases, these tools offer convenient integration with existing workflows and may offer more privacy controls. Here are a few you may find useful:

Adobe Firefly: While Adobe have integrated many AI-based tools into their creative apps, collectively they're known as Adobe Firefly, and are also available in a dedicated web app of the same name. Capabilities include vector generation in Illustrator, Generative Fill in Photoshop, and Generative Extend in Premiere Pro. While AI generation is included with Creative Cloud, the number of credits available can vary widely depending on the plan you're on and your location. In general, using a feature such as Generative Fill costs 1 credit, and plans include between 25 and 4,000 credits per month. Some Premium features (including video generation) use 100–175 credits per second, so many users will face limits and may need to subscribe to an additional plan to use these features. Other non-generative features, such as automatic caption creation, will remain free. You can find Firefly in Adobe Creative Cloud apps and at https://firefly.adobe.com.

Apple Intelligence in macOS and iOS: These features are not the most advanced, but they're free and either run entirely on a device or in a private cloud. Image Playground lets you make low-resolution square images in a limited selection of styles, but several utility AI and automation AI tools can be very useful. For example, text in any image on iOS or macOS can simply be selected and copied as text. The Foundation Models framework built into iOS 26 and macOS 26 offers many more models to allow any developer to integrate generative dialogue, classification, summarization, tagging, and more.

Final Cut Pro: Apple's flagship video editing app includes ML features to isolate voice in audio, create captions, recognize objects in moving video, and create intermediate frames when slowing down video.

DaVinci Resolve: Version 20 of this video editing and color grading application includes many AI tools that run on-device, without a subscription or further costs. Caption creation and voice isolation are just the start; Resolve can be trained on a voice sample and then create new audio in that voice. We'll take a detailed look at these features later in the book.

Avid Media Composer: This established video editing application has integrated PhraseFind and ScriptSync for some time, enabling editors to find shots more easily. In 2025, DeepEditor from Flawless.ai was added, allowing performances to be retimed and visibly spoken dialogue to be changed (even to another language), and for consent to these changes to be gathered from the actors involved. (A separate TrueSync product can work with footage independently.)

Peakto: This image management tool allows you to view and organize images and video across many different catalogs from many different apps. Utility AI now enables image classification, video transcription, and more.

Running your own AI services

As well as AI services built into apps you're already using, you can download new apps to implement AI services on your own hardware. For best speed on a PC, you'll want to use a fast graphics card (GPU), likely from Nvidia; for best speed on a Mac, you'll want a Mac with "Max" or "Ultra" in the name, as these are the models with the most advanced GPU and the most RAM.Why would you want to run AI tasks locally? First, privacy, which may be mandatory when working with assets from some clients. Second, cost, because most of these services are available for free or as a one-off, outright purchase. Third, latency, because you're relying on a local machine rather than "someone else's computer" in the cloud.There are a few ways to run these programs, some of which run like regular apps with graphical interfaces, and others that require you to be comfortable with a command line. As most creative professionals prefer graphical options, that'll be the focus here. You don't need to get your hands too dirty to play with local AI, but you'll have more options for exploration if you can use a command line. A few key local options that aren't part of with standard graphic design, audio, or video apps include the following:

Stable Diffusion: Available for free, to run locally, with many GUI options (https://gist.github.com/AshtakaOOf/d4ee3cc4510dfa1385616eacfcd01652).

MacWhisper: Transcription and dictation with wide format support (https://goodsnooze.gumroad.com/l/macwhisper). Note that PC implementations of the open source Whisper AI are also available.

Picture This: Image generation that integrates with video editing applications (https://apps.apple.com/au/app/picture-this/id6466822042?mt=12)

Draw Things: Free image generation for iOS, iPadOS, and macOS (https://apps.apple.com/au/app/draw-things-offline-ai-art/id6444050820).

LM Studio: The easiest way to run local LLMs if you're not a fan of the command line. On an Apple silicon Mac, LM Studio presents models that take advantage of Apple's MLX frameworks for the best speed when processing ML algorithms on modern Apple hardware (https://lmstudio.ai).

ComfyUI: A node-based tool to connect different AI models together, allowing you to take output from one process and send it to another. Though this isn't a simple process, if you want maximum control over your own system, it's probably the best option (https://www.comfy.org).

Jumper: Local AI-based transcription and clip analysis to enable searching across video and photo media, integrated with common video editing applications (https://getjumper.io).

Strada: This AI tag generation, classification, and analysis tool started with a cloud-first focus, but is shifting to a local cloud model. It promises smart content- and transcription-based analysis of video footage to make it easier to find the right video clip to suit your edit (http://strada.tech).

Summary

In this chapter, we learned a little about where AI started, how these systems work today, what they're good for, and where they fail. If you can discover how AI can help you get more done without compromising your creativity, you'll be well placed to keep creating.Different kinds of AI tools will give you many opportunities to make your work faster, even if you don't want to (or cannot) use GenAI to produce client work. Utility AI will help you find and transform, while automation AI will help you do your existing work more quickly. Not all AI is the same.Though new services appear all the time, different kinds of AI tools will inevitably share strengths and limitations due to the way they're run. For the latest, most capable models, you'll probably want to pursue one or more subscriptions with an up-to-date cloud service provider. Almost all these services offer free trials, so start slow, add additional services if they prove valuable, and keep a close watch on progress across the field.For the cheapest options with the best privacy, you'll need to look for on-device models, but also realize that you may not be running at the speeds or capabilities of the cutting-edge cloud solutions.Between these two extremes, you'll find many apps with outright purchase and subscription options. It's a competitive and quickly changing field.However, AI tools do have some lingering ethical questions hanging over them. Were they made in an ethical way? Does using these tools expose you to legal issues? Before we fully engage with how to harness the power of AI, we need to address several questions regarding the ethics of AI use. For that, read on to Chapter 2.

Further reading

The history of AI. IBM: https://www.ibm.com/think/topics/history-of-artificial-intelligence

Turing test. Wikipedia: https://en.wikipedia.org/wiki/Turing_test

All You Need Is Attention. Vaswani et al: https://arxiv.org/abs/1706.03762

No one actually knows why AI works. Alberta Techhttps://www.youtube.com/watch?v=nMwiQE8Nsjc

AI is creating fake legal cases and making its way into real courtrooms, with disastrous results. The Conversation: https://theconversation.com/ai-is-creating-fake-legal-cases-and-making-its-way-into-real-courtrooms-with-disastrous-results-225080

Everything is a Remix. Kirby Ferguson: https://www.everythingisaremix.info

Diffusion Model. https://en.wikipedia.org/wiki/Diffusion_model

Chapter 2: Ethical Implications of AI Use

Join our book community on Discord

https://packt.link/EarlyAccess

New technology can often be divisive, but it's rarely as polarizing as artificial intelligence (AI) has proven to be in recent years. In some circles, using AI of any kind is seen as poor form, and indeed, I've worked for a client that banned the use of generative AI outright. In some ways, those who are cautious are right to be; there are real concerns about the ethics of AI use, especially in a professional context.These concerns are wide-ranging, starting with questions about the sourcing of training data, and continuing with privacy concerns related to online systems. For instance, can you share personal data with a chatbot, or will that data be used for further training, or even held indefinitely? There are also obvious concerns about the use of deepfakes, and it should go without saying that you shouldn't use AI illegally.Because not all AI-made content is good, there's a broader concern about standards; if poor content becomes more commonplace, does this destroy the market for creative work? This ties into the impact on artists who could be potentially put out of work as a result of AI, as well as environmental impacts from the energy used by the data centers that power many AI tools.None of these issues is simple, and not all AI is generative AI. In this chapter, I'll try to lay out the facts so you can make your own judgment about how to use AI in an ethical way.In this chapter, we'll cover the following main topics:

Copyright

Privacy

Bias

Slop: the quality problem

Human impacts

Environmental impacts

Copyright, fair use, and stolen sources

Although copyright may appear relatively simple, it's actually a complex legal minefield filled with exceptions and loopholes. Publishers of books have had copyright-fueled disagreements with large tech companies before AI training became commonplace; this is not a new fight (https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,_Inc.).However, given that generative AI models promise to create new works in the style of existing works, many authors and artists see AI as an existential threat. Whether that's the case or not, there are questions about copyright that need to be addressed. But it's not totally straightforward.The process of training an AI model involves processing very large collections of text documents or images, and this is the first potential source of conflict—where did the content come from? Computer programs that traverse the web to index or otherwise examine its content are known as crawlers, and one widely used source of text is Common Crawl (https://commoncrawl.org/faq). This collection of documents is taken largely from the web, but it also includes copyrighted works under the "fair use" principle in the US.In the interest of full disclosure, it seems that an illicit copy of my first book, Final Cut Pro Efficient Editing, has already been used as training material for at least one AI system. This was probably inevitable, and I don't expect to be paid thousands of dollars for this training usage, but I'd hope that future LLM training doesn't involve collections of pirated books. A counterpart for images is LAION (https://laion.ai/faq/), a free source of images with alt text descriptions. Although these images are accessible on the web, some are similarly commercial in nature, and, as LAION is for research purposes, fair use exemptions are claimed under German law. While Common Crawl does maintain and distribute the text documents in its training set, LAION does not—it merely links to the original images online. Those who train AI models on these images must download them on their own, presumably at their own legal risk.Of course, other data sources are used, too, and if you place your content on the open web, you can expect it to be scraped for AI training. Although a mechanism exists for marking your content as "not to be indexed" in the form of simple instructions in a robots.txt file, it appears that many web crawlers hungry for AI training data are ignoring these requests (https://www.wired.com/story/cloudflare-blocks-ai-crawlers-default/).A common misconception is that AI models contain all the data they have been trained on, but this is not the case. Instead, a model learns the information presented and the style in which it was expressed. This can be easily verified by the fact that a downloadable image generation model called Stable Diffusion is able to produce art in almost any style, but it's only a few GB in size. The original art isn't copied into the model itself, in the same way that you don't copy a piece of art when you study it.However, for the learning to take place, the content linked by Common Crawl or LAION (or any other source) must first be copied so it can be analyzed, and this process itself may be a copyright infringement. You may be able to view a work of art freely in a gallery, but taking your own photograph of that artwork and reprinting it may not be permitted.As a model is only as good as its training data, obtaining high-quality training materials is desirable, and that has led to some inevitable conflicts. Some generative AI models are able to create images featuring a watermark similar to those found on unlicensed stock images from the Getty Images stock library. It can reasonably be inferred that those images formed part of the training dataset for the model in the first place, and at the time of writing, Getty's case against Stability AI [2] is still underway.In another case [3] brought by Disney and NBCUniversal against generative AI company Midjourney in mid-2025, copyright infringement is alleged because their system can easily generate images of copyrighted characters such as Darth Vader. Complainants have several issues: that these images can be created in the first place, that they are being actively promoted by Midjourney, and that copyrighted images must have been used as part of Midjourney's training data.Indeed, access to training data is a major sticking point between publishers and AI companies. Publishers want to be paid if their content is used for training AI, or to be able to exclude their content from training altogether. Many AI companies argue that asking for permission is too hard, or that "it's all fair use anyway," and want to train on all data that hasn't been explicitly excluded—an opt-out model. Many publishers would prefer the default to be "disallowed" and for training data to be opt-in.Since I am not a lawyer, I'll refrain from drawing any conclusions here, but it is worth noting that some AI models have been advertised as being trained only on permitted sources. One prominent example is Adobe's Firefly dataset, trained on their own stock image and video libraries and used in Adobe Photoshop and Premiere, among others. Tools like this, with data sources that can be proven "safe," may be preferred by some users. Finally, there are questions about the copyright status of work made by AI models themselves. According to a decision by the US Copyright Office [1], work made entirely by AI cannot be copyrighted as it was not made by a human. If you plan to create work that needs to be protected, such as work for a client, make sure a human is involved, at least in part.In addition to questions about how AI models are trained, some tools need to know a lot about you, and privacy must be considered. Let's explore how.

Privacy and personal details

In some ways, privacy considerations are obvious, but in others, they can be more subtle. For example, if you want an AI tool to analyze a database that includes personally identifying information, you'll need to know what's going to happen to that information. Will it be potentially used as part of the training dataset? Could it be scraped or stolen outright? Are you breaking a non-disclosure agreement with your clients by uploading privileged information to an AI tool?We are used to storing all kinds of personal data on our own computers, and have grown accustomed to storing that data on cloud servers as well. Still, asking AI-based tools to look through that data on our behalf is a notable further step, with potentially important consequences.Privacy concerns have led many government departments to ban commercial AI tools outright, and given the risks, they're right to be cautious. Just like web searches, a record of your conversations may be kept, at least for a limited amount of time. If you search for information about a medical condition, could an insurer potentially use this information to exclude you from cover? Could or should law enforcement be able to access files you've uploaded to an AI service for analysis?In mid-2025, OpenAI, the maker of ChatGPT, faced a huge privacy issue when a court ordered it to keep a record of all consumer conversations, though this has now ended. This order was granted in response to a claim from The New York Times that users might be infringing on its content and deleting chats to cover their tracks.Still, a chat where you are in control of what you say is one thing—what about a tool that reads your email or your messages? If you'd like the help of an AI agent that can access important personal data, you'd better make sure there's a robust privacy policy separating your data from the company behind the AI tool. For example, although DeepSeek seems to be capable, the fact that it's located in China gives some Western users cause for concern.Ideally, the company should employ encryption to make sure it never has direct access to your data at all. In theory, because the data isn't available to anyone else, this protects you from both disgruntled employees and external hackers. In practice, because some of the largest AI tools are from companies largely funded by data gathering and advertising, I'm not sure where that privacy curtain will be drawn.One of the key selling points of Apple Intelligence, the AI offering from Apple, is its focus on privacy. Where possible, data never leaves your device and is processed locally, and Apple offers a selection of locally run models that developers can tap into. For more complex operations, data is carefully anonymized and encrypted before being processed on a private cloud server owned and run by Apple, and these workflows can be audited by third parties. It's also possible to use ChatGPT or other models directly, but since this is outside Apple's control, extra confirmation is required before your data is transmitted.In the context of creative projects, be sure to check that a tool you plan to use has a privacy policy you and your clients are comfortable with. For instance, if you're working with video content that can't be placed online, you won't be able to use a cloud-based AI tool. But if your data needs to remain private, make absolutely sure that your uploads aren't used for further training.

Bias, balance, and censorship

Current large language models (LLMs) such as ChatGPT, Claude, and Gemini don't behave like humans do, and remarkably, they're not completely understood—the CEO of Anthropic has said we don't know how AI works (https://futurism.com/anthropic-ceo-admits-ai-ignorance). Some recent studies have shown how at least some parts of their neural networks function, but as these are not rule-based systems, we can't simply add rules to control how they behave. We can make educated guesses, and indeed, bad actors have already shown that they can meddle with internal details to influence an LLM's output—read on for more on how Grok was modified.Inside an LLM, weights are assigned to nodes in a network, and with some effort, it's possible to discover what a particular node represents. With privileged access, researchers at Anthropic located the neurons in Claude that represented San Francisco's Golden Gate Bridge [4] and increased the weighting of these neurons within the model. This modified LLM now responded to most questions—relevant or not—with answers featuring or pertaining to the Golden Gate Bridge. You can find more in the Further reading section at the end of this chapter.Anthropic performed this research intentionally, as part of their program to figure out exactly how LLMs work. However, soon afterward, a clear example of how tinkering could lead to bias was shown by Grok, which started to weave the far-right conspiracy theory of "white genocide" into unrelated answers [5]. While this didn't last long, the cause [6] was someone with internal access changing the weighting on a group of neurons in a similar way.Given that LLMs are somewhat of a black box, and we don't fully know how they come to the conclusions they present, we must be extremely cautious about the information they tell us. Notoriously, the China-located version of DeepSeek will not answer questions about some topics, such as an event in a famous square in 1989, but censorship isn't new.Rather than watching for obvious censorship, it's important to bear in mind more subtle ways in which output is presented. Some LLMs, including ChatGPT, will go out of their way to avoid the appearance of bias and will try to give both sides of a political argument in response to a specific question about one side.I asked about corruption in the right-wing Republican Party in the US and about the left-wing Australian Labor Party. Responses varied, but usually began with a broad statement that corruption wasn't specific to a single party.

Figure 2.1 – Questions about corruption in a political party were immediately broadened

At the end of each detailed response with specific examples of corrupt conduct, ChatGPT offered to provide a list of corruption on the other side of politics.

Figure 2.2 – At the end of the answer, an offer to show the other side of the coin was made

Claude's response was similarly balanced, but didn't offer to give examples of the other side of politics.

Figure 2.3 – Claude's response to a political question

As a final example, Grok answered the question, but in its conclusion, it included specific counter-claims along with examples of corruption on the other side of politics:

Figure 2.4 – The tail end of Grok's response, trying hard to present both sides

Grok's response could be in part due to this system prompt, part of the system that tells Grok how to respond to users: "You are extremely skeptical. You do not blindly defer to mainstream authority or media" (https://www.theverge.com/news/668527/xai-grok-system-prompts-ai). While a moderate amount of skepticism can be helpful in avoiding obvious falsehoods, I'm not sure that building in distrust of all mainstream knowledge sources is the best approach.Despite the claimed attempt at neutrality, Grok has been the most political of all the current LLMs. In July 2025, Grok proclaimed itself "Mechahitler" and gave anti-semitic statements in response to user input (https://theconversation.com/how-do-you-stop-an-ai-model-turning-nazi-what-the-grok-drama-reveals-about-ai-training-261001).Just as different news sources present the same information in different ways, LLMs will sometimes choose to amplify a particular point of view—and of course, each time you ask, you're likely to receive a slightly different result. As Grok rightly says (check the closing sentence in Figure 2.4), it's important to refer to primary sources. References are often given in an LLM's response, so follow those links to make sure that they're real, or verify information another way.

Slop: the quality problem

A key argument against generative AI is simply that it's not very good, and it's often argued that AI art can always be identified as such. While some AI art has obvious tells—humans don't tend to have six fingers or three arms—many images are indistinguishable from real images that have been retouched.

Figure 2.5 – This tree isn't real, but it's… fine? (source: https://pixabay.com/illustrations/ai-generated-christmas-tree-9152917/)

Similarly, though written AI content will often include vocabulary or punctuation that most humans don't use often, with em dashes often being used, AI writing is not always easy to spot. (For the record, none of this book was written with AI, but I do like em dashes, and they're easier to type on a Mac than a PC.)However, though poor quality is an issue with a lot of cheaply made AI content, I'd argue that poor quality isn't a problem that's unique to AI. If you outsource work to the lowest bidder, the quality of the work you'll be given can vary from great to terrible, with or without AI. Free stock art is of similarly variable quality because the gatekeepers on paid stock art sites weed out the garbage. Gatekeepers get a bad name, but without them, we are drowning in content.This problem started before AI became popular, with template-driven design solutions such as Canva. To each user on their own, their design looks fine. More broadly, when all those template-made designs are seen together, or more regularly, the sheen wears off. Humans crave novelty, and though templates work for a while, shortcuts don't work forever. Cheaply-made AI art does tend to have a "look" to it, and it's already putting some viewers off.Because generative AI systems can churn out so much content, so quickly, it's very tempting to use more of it, lowering our bar for "good enough" to send more images to more clients, and write more blog posts for their websites. This may work in the short term, but in the long run, more consumers will associate obvious AI with cheap, low-quality content, and that's not what most clients (or humans) want.As a result, we don't need more content, we need better content—and that's what we should be creating, with or without AI's help. If you do use AI to create something you plan to share with others, it's got to be good, not just OK. Don't be tempted to add to the ever-growing pile of AI slop out there, not simply because it's not good work, but because if clients grow to accept bad work, all our work is devalued.

Human impacts: lost jobs and bad art

While there can be obvious human impacts due to the misuse of AI, such as deporting a person incorrectly flagged by an AI recognition tool or spreading misinformation with deepfakes, our focus here is on AI's effect on artists and creatives.In the creative space, an oft-quoted factor in the use of generative AI is that it will put humans out of work. While numbers are vague for now, we won't really understand the full impact for many years.However, it's important to remember that the capabilities of AI are being oversold to some degree. AI won't be replacing many humans directly, but it will be used to make humans more efficient, and this will lead to fewer humans being employed to do a particular job. In creative fields, AI is replacing human artists in areas where obvious flaws or mistakes can be overlooked.If you want a human touch, you still need to hire a human, and the benefits may not be immediately obvious. For example, if an AI can generate a full feature-length storyboard from a provided script, saving weeks of an artist's time, that's very tempting—even if some of the drawings have flaws. Although that output will be helpful and much better than nothing, it's likely to be fairly generic, as if the storyboard was outsourced with simple instructions.In contrast, an experienced storyboard artist might be able to give creative options for a scene or ask colleagues questions to clarify how a scene will play out, so they can suggest a different approach. The time spent drawing each frame is not wasted, but can lead to valuable conversations that end up creating a better end product.At the end of the day, a production can decide whether it's worth investing in a real artist to create better work, or whether they can make do with a lesser, quicker option. This story has played out many times in my lifetime, and most often, the most convenient option wins, despite its flaws. Here are a few examples:

Phones are more convenient for most people than laptops or desktop computers, even though in-depth research is far easier on a large screen with a keyboard

Cars are more convenient, faster, and can carry more people than horses, even though horses offer companionship

Portrait photographs are more accurate and faster than painted portraits, even though paintings can offer a richer experience

AI is simply the latest example of a convenient technology that largely displaces an old one. And yet, nobody has to use AI, or drive a car, or use a computer, or take photographs. As John Siracusa said in his essay The once and future e-book: on reading in the digital age [7]:

Lather, rinse, repeat. Did you ride a horse to work today? I didn't. I'm sure plenty of people swore they would never ride in or operate a "horseless carriage"—and they never did! And then they died.

While some people do ride horses or have portraits painted, the majority don't value those experiences enough to pay for them. The experiences do still exist, but fewer people use their services. After all, some people still listen to vinyl records and take photographs on film cameras. But it's not the norm anymore.For artists to thrive in a future where AI makes adequate art and writes acceptable essays, they'll need to find clients who care about the difference between OK and great. AI may well have made it easier for people who aren't great at illustration to create illustrations, but AI hasn't made it easier to create great illustrations.That may be a comforting thought for experienced creatives, but what if you're new to the field? If less-critical tasks are being given to AI, how can a trainee be expected to learn the trade and eventually become an expert? The key to learning is usually doing, so if you are new to a task, ask an AI to spell out all the steps to you understand the process; don't just ask AI to do it all for you. After all, you don't get better at something by outsourcing it.Direct human impacts are important, but are there indirect impacts related to power use?

Environmental impacts of data centers

Finally, it's worth examining the environmental hit that AI services could have. Computers obviously use energy, and tasks such as video games, cryptocurrency mining, and AI sit at the top of the pile in terms of energy use. Though individuals are in control of their own energy bills, what about cloud-based services?Currently, we don't have hard data about how much power is being used by the data centers that AI providers use, nor about how much energy goes into the initial training process for AI models. Estimates place a ChatGPT response at somewhere between 1–10x the energy cost of a standard Google search (https://epoch.ai/gradient-updates/how-much-energy-does-chatgpt-use), but that's too wide a range to be informative. Since Google searches themselves now include AI-based responses, that baseline has likely shifted further.We do know that AI processing is more energy-intensive than most other computing tasks, both in training and in use, and that image generation is more intensive than text generation. We also know that smaller, more targeted AI models use less energy than larger, more broadly capable models, that text is easier to create than images, and that images are easier to create than videos. Broadly, you could compare intensive AI use to playing video games—not a large amount of energy for an individual, but a large amount in aggregate.Though there are many variables at play here, using AI models that can run on your own hardware gives you something you can measure, and they're likely to use a lot less energy than a cloud data center. For more complex models run in the cloud, if you want to minimize your carbon footprint, choose a provider with a stated goal of net zero carbon emissions. If you can't find a clear public statement, use a service such as https://ditchcarbon.com to examine the policies of an AI service provider.Currently, Microsoft (100) and Apple (96) rate very highly, Alphabet (parent company of Google) does OK at 67, and Amazon slightly better at 71. Most AI companies use data centers run by Microsoft (Azure) or Amazon (AWS) with stated carbon-neutral power goals, so we can expect impacts to be at least somewhat mitigated.

Summary

While there are questions about some uses of AI, it's possible to mitigate most of these concerns. If you're worried about the origins of a model's training data, either use a model with inputs you can verify or train your own model. If you're concerned about privacy, use a local model or a remote model with encryption and a strong privacy policy.To avoid bias, stay vigilant, and periodically compare one model's answers with answers from other models. Check original sources where possible, and never just trust an AI's output blindly.Slop is a real problem, and if you want to stand out from the crowd, don't use AI (or templates, or other shortcuts) to create quick and dirty work. To keep your own skills sharp, make sure you understand how to do the tasks you're giving to an AI.One approach that may work for many AI-cautious creatives is to use AI only for draft art, for brainstorming, and not for final, finished art. Mood boards and temporary music tracks often draw from existing copyright material—perhaps some of that can be made with AI help?Finally, while it seems inevitable that humans will be affected, it remains to be seen whether the impact of AI will be any greater than any previous technological revolutions. After all, a revolution in education is heralded every few years, but learning new skills still takes time. Finally, keep an eye on the environmental impacts of AI. Though individual actions are small, those actions do add up.Next up, we'll start our focus on utility AI, and kick off with audio.

Further reading

[1] The US Copyright Office on AI: https://copyright.gov/ai/

[2] Reuters' report on the Getty AI court case: https://www.reuters.com/sustainability/boards-policy-regulation/gettys-landmark-uk-lawsuit-copyright-ai-set-begin-2025-06-09/

[3] Ars Technica's report on the Disney/NBCUniversal court case: https://arstechnica.com/ai/2025/06/in-landmark-suit-disney-and-universal-sue-midjourney-for-ai-character-theft/

[4] Anthropic's explanation of Golden Gate Claude: https://www.anthropic.com/news/golden-gate-claude

[5] ABC (Australia)'s report on Grok's "white genocide" claims: https://www.abc.net.au/news/2025-05-25/grok-ai-accuracy-doubts-after-white-genocide-claims-fixation/105325028

[6] xAI's explanation of Grok's "white genocide" claims: https://x.com/xai/status/1923183620606619649

[7] John Siracusa, The once and future e-book: on reading in the digital age: https://arstechnica.com/information-technology/2009/02/the-once-and-future-e-book/

[8] The United Nations on the environmental impacts of AI: https://www.unep.org/news-and-stories/story/ai-has-environmental-problem-heres-what-world-can-do-about

[9] Epoch AI's discussion, How much energy does ChatGPT use?: https://epoch.ai/gradient-updates/how-much-energy-does-chatgpt-use

Chapter 3: Utility AI with Audio

Join our book community on Discord

https://packt.link/EarlyAccess

To kick off the next main section of this book, we'll look at AI tools that perform utility tasks with audio. These tasks may involve some generation, but the focus is on cleanup, analysis, or enabling new workflows—making your job easier or faster. In fact, these tools have made my own video production and post-production workflows far faster and more reliable, tearing up the old rules about what was possible in audio.Today, a video editor can use AI assistance to find their footage faster, enable clients to be more directly involved in the editing process, and fix audio problems that once would have meant a reshoot. While I personally approach these tools from the perspective of a videographer and video editor, podcast production and general audio production can benefit from these tools too.The next chapter will discuss video-focused tools, but here, the focus is audio—even if it's used in a video context.In this chapter, and in the rest of this book, headings will introduce tasks that AI can help with before exploring specific tools and related workflows. Remember that new tools become available regularly, and that the options explored here are not the only ones available. Where possible, tools discussed will be readily available, reasonably priced, and from reliable providers—but we may mention new, experimental tools too.Here, we'll talk about the following:

Transcription and text-based editing

Beat matching

Extracting stems from music

Dialogue recording cleanup

Transcription and text-based editing

Just a few years ago, computer-based transcription—commonly used for captions—was primitive and error-prone. You could use a cheap service to get a transcription that had obvious flaws, or pay a service like rev.com to produce much better results at a cost of around US$1 per minute. Most commonly, transcription was used for finished edits, not source media.Today, an AI system called Whisper, created by OpenAI and released as open source in 2022, is at the heart of many modern transcription services available for little cost and with no per-minute processing fees. While Whisper supports many languages, accuracy may vary depending on the language spoken, and not all systems support multiple languages. For example, caption generation in Final Cut Pro is very fast, but it only officially supports US English at the time of writing.Progress hasn't slowed, and the best algorithms have improved on the original Whisper. They can give excellent results, potentially identifying different speakers and coping with input in multiple languages. I've been pleasantly surprised that transcripts of many hours of speeches usually correctly spell (and capitalize) most place, company, and organization names.However, if any people's names are mentioned, you'll still need to correct those by hand. Almost all names have multiple potential spellings, and a computer can't know whether a name should be spelled "Gerry," "Jerry," or "Geri." While humans can often figure out how a name should be spelled from context (perhaps a name is shown onscreen), a program that only processes audio just takes its best guess.While these flaws serve as a great reminder that a human should still be involved in many AI-assisted processes, correction may not be needed for every application. The captions that accompany a public video should be correct, but creating a transcription of video clips that you plan to edit can make it much easier to find the right words in a sea of source material. In those cases, the benefits of transcription outweigh the flaws.Notably, while many video platforms support automatic captions for videos added to their platforms, they're often not very good. For example, YouTube's automatic captions have been available for many years, and haven't improved. Worse, if problematic captions are burned into the video (such as in a TikTok post or Instagram reel), then typos are saved forever, providing an even more distracting experience for all viewers.Transcription has become commonplace, and you can find support in the following:

MacWhisper: A paid app for Mac supporting several models and many output formats

Figure 3.1 – MacWhisper's transcription of a recent interview, exporting segments to CSV

Adobe Premiere Pro: Lets you enable automatic transcription of every source clip, enabling a full text-based editing workflow

Final Cut Pro: Provides automatic transcription of any timeline, with search through the timeline index

Davinci Resolve: Provides automatic transcription of source clips, with AI IntelliScript able to create an automatic rough cut based on a provided script

Jumper: Provides automatic transcription of source clips for all major video editing applications, to enable quick dialogue search through source clips

Many other applications also provide transcription services, including macOS and the major multimodal LLM providers, so I don't want to focus too much on the specific tools involved. However, the best tool for the job will depend on the workflows you plan to use these transcriptions with.

Hybrid text-based editing workflows

Some uses of AI-based transcription may seem relatively mundane, though they can still be utterly transformative to a workflow in a way that text-based editing within a video editing application cannot. Just a few years ago, documentary workflows involved comprehensive on-set note-taking, logging, cross-referencing, and scrubbing through clips to find the right moments, but today, it can be an easy job for one if the information you need during the edit can simply be spoken aloud.For example, on a recent week-long video shoot, I travelled with a team of three (though I was the only video professional) to interview 27 people at four different locations across Australia. While on the trip, I was able to use MacWhisper to produce transcriptions of all the videos and provide them to the team in a format that could be opened as a spreadsheet. They were quickly able to use colors to highlight the best words spoken for each particular answer, and I was then able to use the time codes as references when marking those sections of each clip.Because each shoot was fresh in our minds, we were able to remember specific people and their delivery style, giving context to our choices. We also employed specific techniques to support a transcription-based workflow:

Begin each video by asking the interviewee to spell out their name. This avoids any issues with speaker identification and spelling.

If you plan to ask multiple interviewees the same question (common in documentaries), then number each unique question and say the number out loud as you ask the question. This makes it very easy to find answers to specific questions in a transcript and allows the interviewer to slightly reword each question if they wish.

Any notes important to the edit, such as "That was a great take!" should be spoken aloud, close enough to a mic to be recorded.

From MacWhisper, export Segments as .csv to separate lines into time-stamped rows in a spreadsheet. Since transcriptions often include commas, this format uses tabs instead of commas to separate columns. It will be automatically formatted if you open it in Apple's Numbers app, but if you prefer Excel, use the Data tab's Text to Columns command to format it correctly.

In any NLE, you can locate a time-stamped moment in a clip with just a timecode, so make sure to create clips that start at 0:00 to make sure that the references your collaborators give you match up to your clips.

A hybrid workflow like this is ideal if content decisions are made by people who don't have access to the video clips themselves, and it'll work in any video editing application. Reading or searching a transcript is much faster than trying to find an answer in a sea of video clips, but be sure to advise your collaborators of the following:

A transcription will often omit "um", "ah," and pauses, so it's not always possible to judge how good a particular clip is just from a transcript

There's often no record of which take might be the best one, though this can be spoken aloud on set

Emotion doesn't come through, and a perfectly spoken line of dialogue may be less impactful than an imperfect line spoken with emotion

Videos cannot be edited as easily as text, so bear that in mind when requesting edits

While these tips are based on an unscripted production, in a largely scripted workflow, AI still has a place. Avid Media Composer offers the option of ScriptSync (https://www.avid.com/products/media-composer-scriptsync-option), which uses AI-based transcription to associate spoken dialogue with a provided script. Rather than searching for words you hope someone said, this tool helps an editor link a script to real-world footage.Hybrid workflows can be very useful, but a full text-based editing workflow can help you immensely if you're a video editor who's empowered to make every decision about the video on your own. The preceding transcription tips can still help, and you may use these workflows to help collaborators inform your editing decisions, but full text-based editing is a different beast.

Full text-based editing workflows: Premiere Pro

In Premiere Pro, make sure you've enabled automatic transcription of all clips. (If you prefer, you can also transcribe clips manually.):

Figure 3.2 – Activate automatic transcription in Preferences

After importing your clips, select a clip containing audio dialogue, then activate the Text-Based Editing workspace from the workspace menu at the top right.The Text panel should become active, and it will include three tabs at the top to guide you through the process. Transcript, on the left, should already be showing you a transcript of the selected clip. There may be errors, as there are in the clip shown here:

Figure 3.3 – The original transcript, good but with a few errors

If you want to correct these errors, you can edit the text just as you would in any word processor, and this is a good idea if you're planning to create captions from the transcript later on. If you'll be generating final captions another way, this doesn't matter so much.

Figure 3.4 – Use the pencil "edit" button above the transcript to correct any errors

As you select the words in the transcript, they'll be selected in the clip or sequence timeline to the right. Conversely, as you play the clip or sequence, words will highlight as they're encountered.To use the transcript for editing, you now have two options. Firstly, if you've added your clip to a sequence, you can select that clip in the sequence, then select words you don't want to keep in the transcript, then press the ; (semicolon) key to Lift (delete and leave a gap) or press the ' (apostrophe) key to Extract (ripple delete, removing the gap) that part of the clip from the sequence.The other option is to double-click the original clip in the project pane, select the words you do want to keep, and then Insert (comma) or Overwrite (period) them into a sequence.With a sequence built, you can continue to use the Text panel if you wish. Pauses are indicated by […], and it's possible to ripple delete them if you wish. Editing with text can be a powerful way to work, and as it lets your clients tell you which words to remove rather than sending you time codes, it can make revisions far easier. Clips can also be searched for words spoken in those clips, but there's no way to search across all clips at once. If you need this functionality, add all clips to a single timeline and search there.Finally, the transcript of the parts of clips that make up your timeline can be converted directly into captions by switching to the Captions tab and clicking the Create captions from transcript button. Captions will then be added to a new Subtitle track at the top of the timeline. Switch to the Captions and Graphics workspace from the Workspace menu.If you wish to burn the captions into the video (recommended only if closed captions aren't available on your delivery platform), then select all the captions and change their appearance using the Properties panel to the right. However, if closed captions are available, it's better to use them rather than burning in the titles, because many people prefer not to see captions, and other people need to control caption appearance themselves.For closed captions only, select the Timeline pane, then choose File > Export Captions and export an SRT file to be uploaded alongside the finished video file. Finally, click the eye icon next to the subtitle track, and they will no longer be included in the final output video.

Full text-based editing workflow: DaVinci Resolve

You'll find all the AI-related features in DaVinci Resolve in the same right-click menu. In Media Pool, select one or more clips, then choose AI Tools > Audio Transcription > Transcribe.

Figure 3.5 – AI Transcription works well in Resolve, but you'll have to initiate it manually

When complete, a new window will appear containing the transcription—and though your experience may vary, I've found it to be more accurate than Premiere Pro:

Figure 3.6 – This unedited transcript contains names such as "Spork Tank Games," which have been spelled and capitalized correctly

To use the text pane to add part of this clip to a timeline, select the text you want to include, then use the buttons at the lower right of the window: Place on Top, Insert, or Append.When you've added clips to a timeline, you can continue to use text to edit by switching the toggle at the top left of the window. The second icon shows you transcription segments that have been added to clips in your timeline, and you can use this to build selections before using commands such as ripple delete.

Figure 3.7 – After switching the toggle at the top left, use this window to select specific moments

If you find that not all clips in your timeline have transcription data, you can use the ellipsis (…) menu at the top right to add it, and also to export subtitles:

Figure 3.8 – Missing clips can be transcribed here

Online text-based editing workflows

While I'd recommend using a desktop-based system for video editing simply to avoid the hassle of uploading source clips, online systems are easier for non-editing collaborators to use. A system such as Riverside (http://riverside.fm) allows clips to be uploaded, automatically transcribed, and then edited by deleting parts of the text that aren't wanted. It works, but it's far less efficient than doing the same steps on a local NLE.

Figure 3.9 – Riverside works well enough, and the transcription is clear

For transcription and editing of specific clips rather than many long clips, I can see the benefit of online platforms. However, the real power of these systems is in automatic editing workflows, and we'll be returning to those later.

Automated audio-based editing workflows

If your focus is on automated text-based editing workflows, I'll be covering those later in the book, in the AutomationAI with Video chapter. There is often some crossover between different uses of AI tools, but as they often do some of the editing work for you, that mostly falls under automation. Similarly, if you'll be translating the captions into another language or generating new audio in another language, that falls firmly under generative AI, and we'll cover those workflows in the next major part of the book.Still, transcription isn't the only trick utility AI can play with audio…

Dialogue cleanup

Most major NLEs now include machine-learning-powered noise removal to enhance voice. While this function has been included for decades in one form or another, before AI became involved, it simply wasn't very good—the results often sounded like an underwater robot.Today, in Final Cut Pro and Davinci Resolve, you'll find a feature known as voice isolation, and it removes anything that isn't a voice remarkably well, with very few noticeable artifacts. While these two voice isolation features aren't identical, they're very similar, and both do a great job of turning recorded dialogue from "OK" to "great."Nearby, both apps also include automatic levelling features: Loudness in FCP, and AI Dialogue Balance in Resolve. These features try to make quieter sounds louder and louder sounds quieter to produce a more balanced result, but they can be blunt tools. However, if you're comfortable with these results, Resolve lets you take this a step further with Timeline > AI Tools > Audio Assistant, which will complete a full balance and mix for all tracks at once.While these features can do a very good job, they're not a complete substitute for a human audio professional, who may make more subtle judgements. Fully automatic solutions always have limitations, but the time saved is a trade-off most of us are willing to make. If you have the time and budget, hire a professional.Premiere Pro has a more extreme dialogue enhancement feature in Essential Sound for clips tagged as Dialogue. Look in the Enhance Speech section, then simply click the Enhance button to process any selected clips. This feature does improve the quality of the dialogue, but it doesn't simply remove the parts of the recording that don't resemble voice. Instead, it's more of a generative feature, trying to make a recording sound like it came from a professional recording booth. To achieve this, it applies much heavier processing, which has pros and cons.If a source recording has severe issues—say, a lapel mic rubbing on a synthetic shirt—Enhance is more likely to produce good results than voice isolation. But there's also a good chance that very faint speech will be turned into much louder nonsense words, because this model is prone to hallucinations.While most of my recordings sound pristine with voice isolation alone, Enhance is a great secret weapon to save the day when everything else fails. If Enhance can't help, you might want to consider a dedicated plugin. iZotope's RX plugins have many useful features, and RX11 includes a dedicated AI repair assistant, which promises to fix all common problems in a single step.For a more hands-on approach, Logic Pro is a full digital audio workstation available as a one-off purchase, and Adobe Audition is a comprehensive audio processing app available as part of Creative Cloud. With or without the help of AI, these audio-focused tools all have tools to rescue poor recordings.With the voice recordings sorted, it's time to look at the music. Sometimes you need to change its length…

Audio remix

Both Premiere Pro and Davinci Resolve offer tools to cleverly change the duration of a piece of music. Rather than stretching the music out or changing its tempo, repeating or similar parts of the music will be identified, and those parts will then be repeated to make the track longer or cut out to make the track shorter. Whatever length you need the music to be, these algorithms are likely to do a very good job and will certainly be much quicker than a human.In Premiere Pro, select the Audio Remix tool; it's the last tool in the third stack in the toolbar. With that tool active, simply drag the right edge of a music track to the left to make it shorter, or to the right to make it longer. Don't be alarmed if the resulting clip ends up a few seconds longer or shorter than the exact spot you dragged it to. Song structure is more important than an exact duration.

Figure 3.10 – Audio Remix in Premiere Pro, showing the repeated sections in a music track that's been extended

Jagged lines on the clip will now indicate where parts of the track have been repeated or cut out. Play over these sections to make sure you're happy with the results. Happily, this feature doesn't cost you any AI generative credits in Premiere Pro.In Davinci Resolve, on the Edit page, select AI Music Editor in the Inspector's Audio tab. (Note that this feature is also available on the Cut and Fairlight pages.) If you know the duration you want, you can edit the time code and click the Adjust button. The 1, 2, 3, and 4 buttons give you different options for the remix, and you can switch between them to pick your favorite. Alternatively, if you'd prefer to work interactively, click the Live Trim checkbox, then drag the right edge of the audio track to the desired length. This feature is in the free and paid versions of Resolve, so if you're using Final Cut Pro or another NLE without a similar feature, Resolve is your friend here.Resolve also offers other AI-based audio tools, found on the Fairlight page. Right-click on a clip, choose AI tools, and you'll find Dialogue Matcher and Voice Convert. Voice Convert is more on the generative side of things, so we'll cover it later in the book, but Dialogue Matcher is easier to classify as a utility. Its purpose is to make one clip sound like another clip, though it's not always able to get completely there.Start by selecting a source clip, then right-click and choose AI Tools > Dialogue Matcher > Capture Dialogue Profile. Next, select a clip you'd like to sound more like the source clip, then right-click and choose AI Tools > Dialogue Matcher > Apply Dialogue Profile. If this is a feature you need, Final Cut Pro has a similar tool called Match Audio, found in the Enhancements menu (a magic wand) under Viewer. Select the destination clip, choose Match Audio, then click on the source clip you want to copy from, and click Apply.With dialogue cleaned and matched, let's look at how to remove vocals (or any other component) from a finished track.

Selecting musical stems

For more complex musical reworking, you may wish to extract the separate parts of a music track to be able to remix them yourself. Sometimes you just need to separate the vocals from everything else; sometimes you want to take out the rhythm section. Depending on your needs, you might choose to do this in Davinci Resolve, in Logic Pro, or with another tool.The easiest solution is in Resolve. In the Fairlight tab, select an audio track, then turn on AI Music Remixer in the Inspector. You can use the simple sliders here to mute or control the levels of Voice, Drums, Bass, Guitar, and Other parts of a track, or access the same options in a dedicated floating window if you prefer:

Figure 3.11 – Resolve's AI Music Remixer is quick, but the results aren't perfect

In general, this tool does allow for some flexibility in the mix, but the separation isn't perfect. For better results, head to Logic Pro, import your track, then choose Function > Stem Splitter. Leave all the options checked on the next page, then wait a short time for the results. In my tests with my own tracks, Logic Pro's outputs are far cleaner than the real-time effect in Resolve. It's also pretty convenient to have each stem available as a separate unit for easier remixing.

Figure 3.12 – Logic Pro's Stem Splitter does a great job of separating components into separate tracks

These established desktop apps aren't the only options today, though. If you don't have Logic Pro, look online and you'll find a number of different options, often making use of the same "Demucs" algorithm. Ultimate Vocal Remover 5 (https://ultimatevocalremover.com) is free, and here are several alternatives worth trying: Moises (https://moises.ai), LANDR Stems (https://landr.com), lalal.ai (https://lalal.ai), and AudioPod.AI (https://audiopod.ai). Some of these services also offer other audio processing tools (such as noise reduction or speaker separation), so consider building them into your workflow if they perform well for you.

Stem splitters can be used as a forensic tool to judge the authenticity of a music track. Because generative AI music has been trained on compressed music tracks, music generated by AI often contains audible artifacts and cannot be separated cleanly into its components. If Logic Pro's Stem Splitter can't get good results, the track you're trying to split might have been heavily compressed, or it might have been made by generative AI.

Whether you pulled the music apart or not, AI can also help you find its rhythm.

Identifying music beats

Let's end on something simple. While it's not usually a good idea to always time the edits in a video to exactly match music beats, it can bring helpful emphasis to key moments. It's not difficult to add markers manually, but you can use the help of AI to spot the beats automatically. In Resolve, right-click an audio track on the Cut or Edit page, then choose Show Music Beats.

Figure 3.13 – Show Music Beats takes a few seconds to initialize, then processes very quickly

Lines on the clip will now show potentially impactful places to place your edits.For the same trick in Final Cut Pro, look to BeatMark 2 (https://ulti.media/beatmark-2/) from Ulti.Media. You can process a clip in this third-party app, then export an FCPXML (with markers added) back to Final Cut Pro.

Summary

Utility AI is undeniably useful, and though it's not perfect, it's a lot better than nothing. Older audio algorithms could get the job done, but a little machine learning has added a lot. Of course, if you're an expert, you'll be able to spot the flaws in their work, but if you're an expert, they're just a starting point for something better. As ever, if the results are good enough, you can use them on their own, but if you also want to get better at a task, tweak some settings, listen more closely, and you'll get better at that task yourself.In the next chapter, we'll take a look at how Utility AI can help you with image and video production tasks.

Further reading

So It Begins... Is This A Real Band Or AI?: https://youtu.be/3Nlb-m_vKYM?si=jDq1gYu15dlYsbks

Chapter 4: Utility AI with Images and Videos

Join our book community on Discord

https://packt.link/EarlyAccess

Images and videos drive much of the creative space, so you'll find many opportunities for AI to help organize, classify, and select parts of images and videos. Some of these Utility AI tasks will sometimes stray into the Generative AI space, but if they're covered in this chapter, these features will create new work that is very much based on existing work, instead of creating something entirely new.Automation AI, covered later in the book, is more about replacing your existing workflow, rather than augmenting it. Utility AI in this context is more about helping you find and select, rather than helping you create or doing your job for you.If you shoot photos or record videos yourself, you'll already know how hard it can sometimes be to find a particular shot or clip. Personal and family photos eventually grow into a huge, unmanageable pile, and just like printed photos in a shoebox, if they're not organized, they'll rarely be seen again.Cameras do create metadata (like time, date, and location) which can help to locate a photo later, but they can't find the photo where your cousin was wearing that shirt, or the video of the sheep scratching itself on a fence. For that, and for all the times when a client asks for that shot with the thing, AI is the best solution we have.Once you've found the shot you need, AI can help some more, selecting the person in the frame and letting you color-correct them separately. You can automatically convert a widescreen video into a portrait one, and get AI to reframe all the shots for you. When the client sends you time-stamped feedback, you can even ask AI to annotate your timeline to make your editing process smoother.Lastly, by straying slightly into the generative space, you can use AI to help with conversion of photos and videos from monoscopic to stereoscopic — and even beyond.Though images are videos are treated separately in other parts of the book, in the Utility AI arena, both are covered together. Several tools allow you to work with both still and moving images in the same way, and it didn't quite make sense to separate them.Here, we'll talk about:

Organization and classification for photos and videos

Selecting people and objects

Reframing for aspect ratio changes

Stereoscopic conversion

Interpreting editing requests

Removing jump cuts

Retiming video

Upscaling images and videos

Organization and classification for photos and videos

Searching for a photo or video can be time consuming if you can't remember when or where you captured it, and AI can provide powerful ways to make complex searches easier. Note that some tools go further than search, performing aspects of your job for you: either automatic color correction, automatic logging, or outright image culling. As these tools (including Eddie.AI, Aftershoot and more) focus on not just organization but on automating your workflows, they will be covered in the Automation AI, Part 4 of the book.Most photographers use a Digital Asset Manager (DAM) to organize their photos, and some of these tools can also manage videos. Conversely, while most users would use a video editing program to manage only videos, these apps can also be used to manage still images. You may find that AI tools are better in some of these programs than others, so consider keeping an open mind when it comes to AI search.We'll start with a simple solution that most Apple device owners will have encountered.

Apple Photos

Apple Photos is a basic solution to photo and video management, and although it may not scale to the needs of most creative professionals, it does come with comprehensive AI tools that many overlook. A photo taken on an iPhone will be automatically tagged and can be found by searching on its content, and this is all thanks to AI, but this feature does not rely on the space-limited and online-only iCloud Photo Library.If you have an iPhone handy, head to the Photos app and click the magnifying glass to search. Rather than typing something that metadata recorded like a date or a location, search for content — beer, or dancing, or green. Your phone will find all the photos containing that subject, or activity, or color, or even that word written somewhere in an image or video. Yes, you can find a photo of a sign by searching for the text in the sign.Amazing and useful though this is, if you're not loading all your images into Photos, you won't be able to make use of this feature. Since Photos isn't the best place to edit videos, it's unlikely you'll store or access most video clips there, and you may prefer to use another app like Lightroom to manage client photo shoots. Still, there are many ways to access powerful AI-powered search, and we'll start with Photos.Though it's not obvious, you can create as many Photos libraries as you need, and use one for each client or for each job if you wish. You can also load images and videos into Photos without duplicating them, or even moving them from their current location.

To set this up, hold the Option key as you launch Photos, create a new library by pressing the Create New button, and store it on the same storage device as your images or videos. Next, head to Settings and uncheck Copy items to the Photos library. Finally, you can drag in your images and/or videos from their current location. (You can repeat these steps and choose your original Photo library to return to it later, and you may also wish to re-check the "Copy items to the Photos library" checkbox.)It's perfectly OK to drag in images from a Final Cut Pro Library, Premiere Pro or Resolve projects, or from a Lightroom CC library — you'll just be using this for search. Unfortunately though, AI analysis isn't immediate, and worse, you can't force it to take place. If you enter text in the search field and the pop-up below says Indexing then you'll just have to leave Photos running until that process finishes. This shortcoming may make Photos the wrong choice for quick turnaround client work, but it may still be appropriate for longer-term and personal jobs.If you can wait, searching for the content of an image or video will help you find things much more easily. If you want more, you'll need to look for another app or plug-in with similar powers — and there are many to choose from.

Excire Foto

One of a few local digital asset management tools that incorporate AI features, Excire (https://excire.com/en/) lets you search by text prompt, such as girl surfing a wave or autumn leaves in a forest. Similar to Apple Photos and Google Photos, if you tag a face with that person's name, you can find other photos they appear in.Excire Foto is an independent app, but Excire Search is a plug-in that adds prompt-style search to Adobe Lightroom. The advantage of using a free text-based search is that you're more likely to find the photos. Here, I've searched for yosemite on a collection of free stock photos:

Figure 4.1 – These images are all judged to be Yosemite-ish

Though none of these images have "Yosemite" in their names, AI is able to match these as either taken in Yosemite National Park, or looking somewhat like it. Excire scans and searches quickly, though some searches (such as water) didn't return all possible images.

Peakto

Uniquely, Peakto (http://peakto.com) doesn't force you to change how you organize your photos or videos. Instead, it ingests your assets from their existing libraries, including Apple Photos, Adobe Lightroom, Capture One or Aperture, and videos managed by Final Cut Pro, Premiere Pro, or DaVinci Resolve. Once ingested, AI processing classifies and categorizes your assets, allowing you to search across your entire digital life, or just across a single library. This is the only tool I've used that can show me every photo I've ever taken of waves all at once.

Figure 4.2 – Many hundreds of shots of waves, pulled from hundreds of GB of images

As well as enabling content-based search for your own requests, Peakto categorizes your images and groups them by content, color, brightness, saturation, and contrast, so if you need a predominantly green shot, you can find it relatively quickly.Peakto recently added more comprehensive video support, extending the AI search capabilities to video clips. You can find clips that are similar to an existing clip, search across multiple libraries with natural language, add keywords automatically, and enable workgroup access to a local device. Additionally, audio is automatically transcribed, so you can search through spoken dialogue too.In my own tests with many thousands of photos and videos loaded, Peakto's AI search did make it easier to find specific items. While only a few of my own jobs span multiple years, if you're working on longer-term projects, Peakto's AI search could help you find a needle in a haystack. Transcription search through a large and growing collection of video clips allows you (or a non-editor collaborator) to find every instance of a specific spoken word, potentially across many years of footage and many separate projects. Content recognition in videos is also effective — here, I was able to find all the clips in a single FCP library that contain statues.

Figure 4.3 – Each of these video clips does indeed contain at least one statue, and now they're easy to find

Transcriptions aren't as accurate as those found in some other apps, but they're good enough to be useful. Dialogue search is something that could be done more slowly with a transcription search in an NLE, but multi-library search is a unique benefit of Peakto that goes beyond AI.

ON1 Photo Keyword AI

ON1 Photo Keyword AI (https://www.on1.com/products/photo-keyword-ai) is available as a standalone app, or as part of ON1 Photo RAW. After using AI to detect content-based keywords, it then embeds these keywords using industry-recognised XMP metadata, and is compatible with popular photo management solutions like Lightroom CC.While this solution doesn't offer the aesthetic judgements that Peakto does, its results can be seen in other apps. Keywords automatically generated in this app can be accessed when imported into Lightroom CC and other DAMs, so you don't necessarily need to change your workflow to integrate AI.However, one shortcoming of this app is that the keywords generated may not be good enough to help you find an image you're looking for. This can be an issue when an AI generates a specific list of keywords; if you want to search for Yosemite but the AI only generates mountain or landscape then you're out of luck. Instead of typing any phrase you wish, you'll need to look through the list of generated keywords.

Figure 4.4 – ON1 Photo Keyword generated many keywords, but not "Yosemite"

This app offers a free trial, so you can try this app with your own media, to see if it could help your workflow. However, some AI-based engines take another approach, not simply directly based on keywords.

Jumper

A plug-in that integrates to video editing applications (including Premiere Pro and Final Cut Pro with more to come), Jumper (https://getjumper.io/)can perform AI analysis across any assets shown to it, then enable search across them all at once. As it's primarily designed for video, Jumper won't just find a clip, but a specific range within a clip that contains the item you're searching for. This tool does work with still images, though you'll probably want to load them into your NLE to make it easier for Jumper to see them.

Figure 4.5 – Jumper does need to be told which media files to index, and if you want to index video, audio or both

Instead of generating a limited set of keywords that stay with the images, Jumper uses a fuzzy search. It can find the correct mountain image shown earlier after a search for Yosemite, even though that information appears nowhere in the metadata for that image.The downside of this fuzzy approach is that Jumper always finds something. If you search for elephant in a group of images that definitely don't contain an elephant, you'll still be shown something. Still, given that everyone will search in a slightly different way, Jumper's fuzzier approach is more likely to produce useful results more quickly. While the initial processing can take a little while for videos with audio, search is nearly instant.

Figure 4.6 – Yes, Jumper can find "Yosemite" images with its fuzzy search

Because videos and their audio are both indexed, you can search through visuals or the words spoken in dialogue. While you can't access a transcript directly, you can search for a word or phrase to locate it quickly, and then add that specific phrase to a timeline.

Figure 4.7 – This dialogue was all correctly transcribed and the correct phrase was found, along with similar phrases (thumbnails blurred for privacy)

Strada

Early demos of Strada (http://strada.tech) placed a heavy emphasis on tagging and analysis, and it's able to assign keywords to specific parts of clips where objects are recognized. These selected ranges can then be sent back to your video editing app of choice, though FCP is the tool best suited for range-based keywords. Transcription is supported so you can search by spoken dialogue too, and translation is also built in — it's comprehensive.

Figure 4.8 – An early shot of Strada during development, showing how it's able to detect objects at specific times

This tool is still in development, and has shifted focus from cloud-only to local-first, so it remains to be seen if this feature will return. Still, check this one out if you need AI keyword support for video.

Axle AI

Axle AI (https://axle.ai) is an AI-powered video automation platform that incorporates an AI-based tagging and search feature called Axle AI Tags. While we'll return to this platform in Chapter 11: AutomationAI with images and video , tagging could be useful on its own.

Figure 4.9 – Axle AI Tags performs analysis to figure out what's in each section of a clip

Axle takes a slightly different approach from Jumper or Strada. Where Jumper doesn't expose its search index, and Strada finds single words, Axle uses AI to describe what it sees in a shot, then allows you to search through that text. People and objects can be recognized, and this categorization can happen locally, on premises, not in the cloud. While it's a more expensive solution (US$200/month) than Jumper , it may still be worth considering as a workgroup solution.

Other options

Adobe Premiere Pro Media Intelligence

Introduced in the Adobe Premiere Pro (https://www.adobe.com/products/premiere.html) 2025 Beta release this feature automatically classifies media you've imported into your project and allows you to search for content with natural language. Though Premiere Pro is of course most suited for video assets, if you want to use this feature to search through photos, you can import them and search them too.Unfortunately, the analysis I've tested to date isn't as strong as Peakto, Excire or Jumper; searches for Yosemite came up short, though keywords added in other apps were successfully imported here. If you're searching for dialogue, the full text-based editing workflow we covered last chapter is excellent, but there are better options for content-based search. Jumper is available for Premiere Pro if you need the functionality, though.

Google Photos

Many of the same "smart" features that Apple Photos offers are available in Google Photos (https://photos.google.com/) too. Searching by content works well, faces can be recognized, and mobile apps are available for quick upload from those devices. However, if you're a professional photographer or videographer, I'd think twice about using an online-only solution.Even if you're usually able to connect to the internet, at some point your connection will be slow or non-existent, and your workflow will collapse. RAW photos and original video clips are too large for many creatives to upload in bulk, so online-only solutions won't work for everyone — but this might be fine for your personal phone-based photos.

Photoprism

This open-source solution (https://www.photoprism.app/) runs in a container, performing AI classification and allowing you to search using natural language. While the AI search doesn't seem as flexible or powerful as some of the other solutions here, the free, local nature of this product means it might still find a use for some applications.

Monument

Available in cloud (Monument Cloud) and self-hosted (Monument 2) versions, Monument (https://www.getmonument.com/) offers an alternative to iCloud or Google Photos, or indeed to storing your images in a cloud at all. In either cloud or local forms, it acts as a platform- and device-independent photo storage system, performing AI classification to make searches easier. While I can see the utility of a solution like this in a family environment, I'm not sure that many professionals want to keep their high-resolution original files on a third-party hardware device or upload them all to a cloud.

Selecting people and objects

Selection is at the core of retouching and photo improvement tasks, and AI-based area selection has improved to the point where workflows can be reinvented. Just a few years ago, a photographer might have to spend significant effort adjusting the light illuminating a person and the background behind to get the correct degree of subject separation.Modern segmentation algorithms are now good enough to select not just a person quickly and accurately, but separate parts of a person; even separate parts of an eye. This allows a photographer to adjust light on the person and the background independently without extensive manual selection, and lets them work faster.These features have been added to various photo-focused apps on various platforms, and with varying degrees of finesse. Many Apple-made tools include automatic background removal, and though it's easy to access, it's not controllable enough for most professional tasks.ML-powered subject selection appears in apps like Pixelmator Pro, and while this feature is helpful in portrait photography, other tools offer more extensive selection options. At time of writing, the most complete set of masking tools is found in Adobe Lightroom Classic, so let's start there.

Lightroom Classic and Photoshop Masking

In recent versions of Lightroom Classic (https://www.adobe.com/products/photoshop-lightroom.html) the masking functionality has expanded to automatically select people in an image — one, more, or all the people in a shot. In the Develop mode, on a photo containing people, click the Masking mode icon below the Histogram. At the bottom of the Masking section, people in your photo will be recognized, and you'll then be able to click on a person's face to select them, or part of them.

Figure 4.10 – In this illustration, all seven people can be selected individually, or all at once

A magnifying glass on a black background AI-generated content may be incorrect.Quick tip: Need to see a high-resolution version of this image? Open this book in the next-gen Packt Reader or view it in the PDF/ePub copy.

The next-gen Packt Reader is included for free with the purchase of this book. Scan the QR code OR go to https://packtpub.com/unlock, then use the search bar to find this book by name. Double-check the edition shown to make sure you get the right one.

Of course, most images stored in Lightroom are photographs of real people, and image correction tasks can be quite specific — brighten the eyes, change the color of a top. Selecting these items was once only a task for Photoshop, but now it's possible in Lightroom too.Within each recognised person, you can now select the entire person, or dig deeper and select just part of them: different areas of skin, eyebrows, parts of their eyes, lips, teeth, hair, or their clothes.

Figure 4.11 – Selections like this allow you to fine-tune a person's appearance without Photoshop

These AI-powered selections can also recognize parts of a landscape, automatically segmenting skies, mountains, vegetation, water and more.

Figure 4.12 – If you select Landscape instead of a person, you can then choose which part of a landscape you wish to select

With a mask selected, all the standard Lightroom controls become available, allowing you to correct the color, exposure, sharpness and more within just that part of the image. Tasks that once took significant effort can now be done nearly automatically, in a preset.The most complex tasks still require a trip to Photoshop, and we'll spend more time there in the Gen AI section of the book. You'll also find these automated selections in Photoshop, though they're hidden a little deeper. Open an image, choose the Object Selection tool, and you'll see the same selection options (people, eyes, etc.) in the Options bar at the top of the screen.

ON1 Photo Max Masking

Comprehensive smart selections aren't only found in Adobe's apps; ON1's photo apps can automatically segment parts of images too. While the Inspector offers some controls which automatically affect only part of a face, if you select an image without people, you'll be able to select specific parts of that photograph with similar results to Lightroom.Figure 4.13 – Selecting "flora" gets most of the trees, but the selection near the sky is fuzzy

When processing an image, local selections can be accessed in the Develop tab, where you can choose exactly where to apply an automatic Brilliance AI effect, and also how strong an effect you want to use. For more control, use the Local tab to add an Adjustment, then look to the floating Properties panel and pick a region in the Masking section. Finally, you can control the correction in the Inspector to the right.For more precise control, you can also use the Super Select AI tool (a magic wand icon) and then simply click on an object to select it. This works for clearly defined objects, and also to select parts of a person like their hair. These masks can be adjusted by painting, but fine details around a selection's edges are difficult to accurately select by hand.In my testing, Lightroom Classic produced masks with more clearly defined edges, and this can be very helpful around objects like trees and hair. The less precise masks in ON1 Photo Max meant that the edges of complex areas became visible after adjustment, and I couldn't push my corrections as far as I'd like to. Tools are included to adjust a mask, but that takes time, and they aren't always enough.These two apps aren't the only ones with image segmentation advances; I'd expect this tech to spread into other image processing apps in the near future. You'll also find similar tech in some video apps, starting with…

Final Cut Pro Magnetic Mask

While automatic image segmentation is very helpful for still images, it's almost a necessity for moving images. It's just not feasible to manually adjust a selection 24 times every second! Before AI-based selection algorithms, tracked selections were either painstakingly tedious to create, or used large fuzzy edges to disguise their imprecision.With Final Cut Pro 11 the Magnetic Mask was introduced, allowing selection of many kinds of objects — even as they move around a shot or are occluded by objects moving in front. You start by dragging any effect or color correction to an object or person in the Viewer, and wait for the target to highlight.

Figure 4.14 – Your author, prepped for color correction in Final Cut Pro

Selections can be fine-tuned with painting if needed, and then an entire clip can be analyzed. Additional Magnetic Masks added to the same clip will appear in different colors, and many kinds of objects can be recognized and tracked.This feature makes the impossible possible. I've successfully corrected a person separate from the (much brighter) projection screen they sometimes walked in front of, in a single continuous track, in a 50 minute clip. It's really very good. This builds upon (and is far more effective than) an older macOS feature (used in plug-ins like FxFactory's Keyper) which could automatically recognize people in a shot.Still, all the major NLEs like to copy features from one another, and FCP isn't the only one with this trick…

DaVinci Resolve Magic Mask

In Resolve, the AI Magic Mask feature allows you to select which parts of an image you wish to correct. In the Color page, select the Magic Mask icon from the central panel, then adjust the controls below to increase quality to Better.

Figure 4.15 – The AI Magic Mask controls with part of a clip tracked

With the settings adjusted, and starting on the first frame, click repeatedly on the image to identify which parts of the frame (for example, a person) you wish to correct separately. To see the mask, quickly adjust the exposure (or anything else) on the Primaries wheels below, or use the Toggle Mask Overlay button at the top right of the Magic Mask controls in the centre. If any unwanted areas have been selected, hold option or alt, then click on those areas to remove them.

Figure 4.16 – Though it wasn't visible at first, the hand here was automatically added to the tracked area around the person

Finally, track the selection throughout the clip using the Track Forward button. It's also possible to Track Backward if you didn't start at the beginning of a clip. Because this tool knows what people look like, if a previously unseen part of a body pops into frame — for example, a gesticulating hand — this will be automatically added to the selection.This tool isn't as fast as FCP's Magnetic Mask, but it's controllable, effective, and flexible. If you want to take this further, Resolve has many amazing tools for color correction, including the Relight effect which can do an amazing job.What if you're using Adobe apps?

Premiere Pro Object Masking

It's taken a while, but the beta version of Premiere Pro finally added an Object Masking tool in September 2025, and it does much the same job as the tools in FCP and Resolve. To use it:

Add a clip to a sequence, then select the clip and choose the new Object Masking tool, second from last.

Wait a moment for your clip to be prepared, then hover over a person or object in the Program monitor and click on it. The selection should be accurate, but you can use the + and + buttons to finesse it if needed.

Open Effect Controls and look for the new Object Mask setting on the clip. Press the button in the middle of the Tracker controls to track in both directions from the current frame. A dialog box will show progress, which should be fairly quick.

When the track is complete, you can use this mask for color correction by switching to the Color workspace and then making a change in the Lumetri Color panel to the right. To instead mask out the background, right-click on Object Mask and choose Use as Opacity Mask.

Figure 4.17 — Premiere Pro Beta's new Object Masking tool in action

This tool will be received warmly by Premiere Pro editors. Those using After Effects have another option with more control, though.

After Effects Roto Brush

Import your clip and create a composition to hold it. In the Timeline window, double-click your clip to load it in the Layer panel, then click the Roto Brush in the tool panel at the top.Next, add paint strokes over the object you want to keep. If the selection includes areas you don't want, hold option or alt as you paint over those, to remove them. The effect will automatically propagate through the rest of the clip if you play it with the Space Bar. Like the previously discussed tools, the modern Roto Brush is smart enough to add a limb that enters the frame to a selection of a person.

Figure 4.18 – This arm wasn't in the initial selection, but has been added automatically

Speed isn't as fast as FCP's Magnetic Mask, but it's a little faster than Resolve's Magic Mask. Built-in features aren't the only way to use AI in video, though — well-known plug-ins are now integrating AI features too.

BorisFX Mocha selections

BorisFX Mocha has been used for complex video masking tasks for many years, and as of mid-2025 it now includes AI assistance for the creation of the initial mask. Rather than using the traditional Bezier tools, a new tool allows you to create a new Mask ML Layer. Clicking on the head of a person, followed by a second click lower down the body, is all that's now usually needed to obtain to good selection — a massive time saver over manually drawn paths.In addition to click-based selections, you'll also now find a prompt-based selection method. It's not always perfect, but typing the name of object(s) that you'd like to select, like cars or people, is a good step forward.As the algorithms are now well-documented, I suspect many other tools will incorporate AI-based image segmentation. Even the most basic Subject Selection feature in Adobe InDesign's Text Wrap feature can save you a trip into Photoshop, and in this context, the results don't need to be pixel-perfect to be effective. Watch for more image segmentation features in your favorite apps sooner or later.

AI color correction

While automatic color correction tools have been present in many video and image editing applications for many years, you may not realize that some of these are based on machine learning algorithms — even some of the earliest Auto corrections in Photoshop.

The more recent options, including the automatic options in Final Cut Pro's Color Adjustments, are generally more useful than earlier adjustments, but none are perfect. While some of these features are useful as a starting point for further correction, I'm yet to see a magic "make it look great" button that works every time.

If you want a reliable solution for consistent, serious color correction, test out a solution like Colourlab AI (https://colourlab.ai), which supports many video editing applications.

AI-powered subject selection also helps with reframing when changing aspect ratio — let's take a look.

Reframing for aspect ratio changes

With the rise of vertical social media videos, aspect ratio changes (usually from widescreen to portrait) have become more common. Converting between these formats isn't always trivial, and I recommend shooting wider than normal, zooming in on the widescreen version and cropping the edges of the portrait version. If your camera allows you to shoot "open gate" then you can simply crop that taller frame (4:3 or 3:2) to widescreen and to portrait independently.However you shoot, each clip in an edit will need to be adjusted when transitioning between aspect ratios, and all the major non-linear editors (NLEs) include a feature to make this smoother. Let's explore each.

In Final Cut Pro:

right-click on a project (i.e. a timeline) in the Browser.choose Duplicate Project As.

In the subsequent dialog, choose the new aspect ratio (likely Vertical) and a resolution, then check Smart Conform. Each clip will be repositioned, using machine learning to pick the most important part of the shot.

Play through the sequence and adjust position and/or scale if needed. Note that though you can add animation yourself, FCP won't track movement for you; it picks a single position and sticks to it.

In Premiere Pro:

Right-click on a sequence, then choose Auto Reframe Sequence.

In the dialog that appears, choose your new aspect ratio, likely Vertical. Note that the new sequence uses the name of the current sequence with the new aspect ratio in parentheses, such as Sequence Name (9x16), and it will be created in a new bin (i.e. folder) called Auto Reframed Sequences.In this bin, find the new sequence (with the aspect ratio in the name), right-click, then choose Sequence Settings. Check that the Frame Size meets your needs — because Premiere only adjusts the width of a sequence rather than exchanging width and height, and you may prefer to use 1080x1920 for vertical videos.

Play through the sequence and check that each clip looks right. While Premiere will adjust framing within a shot, this animation isn't always what you want, and the easiest fix is to turn the Auto Reframe effect off.

To do this on each clip that needs it, find the Effect Controls panel, then click the "fx" icon next to Auto Reframe. To reset the position, right-click that clip in the timeline and choose Fill to fill the frame. In Effect Controls or Properties, adjust scale and position if needed.

In DaVinci Resolve (Studio version):

Right-click a timeline and Duplicate it.

Rename the new timeline with the target aspect ratio.

Right-click and choose Timeline Settings.

Uncheck Use Project Settings in the lower left, then choose Custom in the Timeline Resolution menu at the top.

Type in the desired target resolution.

In the Mismatched Resolution menu at the bottom, choose Scale full frame with crop. In the Inspector, in the Transform section, look for AI Smart Reframe at the bottom of the section. Within this section, leave the Object of Interest set to Auto, and press Reframe to track throughout the shot.

If this isn't successful, change Auto to Reference Point, then move the box that appears onto the object or person you want to keep centered in shot.

Press Reframe again, wait for processing, and that area will stay in the middle of your reframed image.

No matter which tool you choose, you'll be able to get at least some assistance with reframing.

Stereoscopic conversion

Stereoscopy, the capture of stereoscopic photographs, has been popular for over a hundred years, and stereoscopic video has been popular (on and off) since the 1950s. To capture stereoscopic photos or videos, normally a left and right image must be recorded at the same time, either using a camera with two lenses, or two synced independent cameras.However, most images are obviously captured in 2D. Because viewers using a device like an Apple Vision Pro can see images in 3D, machine learning algorithms have now made it possible to turn a 2D image into 3D. This involves image segmentation and depth recreation, to decide which parts of the image are closer to the camera and which are further away, and also a generative component, to fill in any missing areas behind foreground objects.

Figure 4.19 – This video, shot in 2D and converted with Owl3D, has impressive depth that's hard to share on a 2D page

In the latest visionOS, it's possible to create not just a stereoscopic version of the image, but a 3D version of the image with freedom to move your viewpoint in any direction. This is known as a Spatial Scene, and so far, it's even more effective than the previously available Spatial Photo conversion. Currently this is built-in to the OS, but not something that's available to common third-party apps.Third-party apps can transform video from 2D to 3D with varying levels of success. The best results I've seen myself have been with Owl3D (which offers Mac and PC apps) but it's possible to perform live conversion at lower quality with apps like Moon Player, direct on the Apple Vision Pro. While the output from these apps isn't perfect, the quality of 3D conversion is likely to continue to improve. Today, if you want to create a stereo image, it's best to capture in stereo, but watch for improvements in this space.

Managing time-stamped editing requests

During video (and audio) production, it's common to receive feedback requests, asking for specific words or time-stamped areas to be removed, for an alternative take, for a shot to be replaced, or an effect to be changed. But if a request affects the length of a shot, it can make other requests more difficult to interpret. For example, if a 5-second shot is to be removed at the start of a timeline, the time stamps of all other requests after that point are now invalid, and difficult to follow.One old-school way to solve this problem is to tackle the list backwards, starting at the end, so that earlier timecodes in a change list remain valid. But this doesn't always work well; sometimes one request relates to an earlier change, and context means it's usually easier to edit from start to finish. What's the solution?Markers provide a great way to solve this issue, allowing you to add a virtual sticky note to every spot on a timeline where a client has asked for a change. Markers exist in all common NLEs, and the shortcut key M will add a marker to a timeline clip in Resolve, Final Cut Pro and Premiere Pro. Because a marker is attached to a clip and not just a timecode, markers will move when clips are deleted from a timeline, so you can tackle a change list from start to finish.Adding all those notes by hand can be a little tedious, though, and if you use Final Cut Pro, the app Marker Toolbox can use AI to process a client-friendly list of timecodes into to do markers with instructions attached. (Marker Toolbox can also convert comments from online review sites like Vimeo and Frame.io to timeline markers.) This gives the best possible experience for an editor and for their client, and they just need to give their notes a very loose structure. For example, they could write:

5 secs end this shot here

0:35 replace this shot

0:47 cut this section from here

0:52 end cut here

1min 23sec fix spelling in title, should be "Jeri"

1:53 check long blank section at end?

Marker Toolbox, packaged as a Final Cut Pro workflow extension, can turn all that into sensible markers on an editing timeline very quickly. If your clients can follow simple rules for writing text like the example above, it's going to work just fine. If the requests are less well formatted, AI processing from ChatGPT (or a local LLM) will make sense of them.After installation, you'll need to press the plug-in icon to access installed Workflow Extensions, then choose Marker Toolbox. Press Settings (below the text areas) and make sure the frame rate in this dialog matches your timeline.

Figure 4.20 – Rough instructions on the left become accurate time-stamps on the right, ready to drag into a timeline

As shown in Figure 4.19, paste your client's feedback into the left-hand text pane, then press Process Comments Locally below. You'll now see the processed change list on the right, and if the client expressed themselves clearly, it should be fine. If not, you can reprocess with ChatGPT by adding an API key (accessible in a paid account) into settings and then pressing Process Comments with ChatGPT. The developer plans to add support for Apple Intelligence local processing soon.Assuming you now have good results on the right, drag the green icon on the lower right to the start of your timeline in Final Cut Pro, visible behind this window.In this example timeline, markers are placed onto titles, each containing the same feedback, all within a new compound clip containing a timecoded stream. To enable the markers to flow along with changes, select the newly added clip, then choose Clip > Break Apart Clip Items. Finally, delete the Timecode track that's created.

Figure 4.21 – The first marker in the list can be seen in the timeline — note that the title connects to the clip and the marker is on the title

Markers (with accompanying titles) now connect directly to the clips below them, and will stay connected to the right clips as changes are made. All markers can be seen in the Timeline Index, to the left of the timeline, in the Tags section. This workflow doesn't use a lot of AI, but it makes a common task easier, and allows clients to express themselves more freely.Another feature clients will enjoy is removing their mistakes, which you can sometimes do without leaving a trace.

Removing jump cuts

Hiding an edit in plain sight is a feature is common in editing applications today, and though it's not part of the modern AI toolset, it's still powered by machine learning.In any major NLE, you'll just need to place a talking head video on a timeline, then remove part of it to create a jump cut. You can do this by selecting two separate ranges in a source clip, and then adding them sequentially to a timeline, or by using the Razor or Blade tool, making a couple of cuts, then selecting that section in the middle and performing a Ripple Delete.At the end of the day, technique is up to you, but you'll want to end up with two clips in which the same person is in roughly the same position. Ideally, the two shots should be as similar as possible: eyes open in both, head facing the same direction, and so on. Avoid situations where part of their body (such as a hand) is visible in one shot but not the other, or you can end up with a mess like this:

Figure 4.22 – Flow, Morph and Smooth Cut will all fail if you try to blend two frames like this, with different hand positions

Now, you'll need to find the right transition. In Final Cut Pro this is the Flow Transition, in Premiere Pro, it's the Morph Cut; in DaVinci Resolve it's the Smooth Cut. Apply the transition, wait for processing, then play it back. The transition will use machine learning techniques to morph one clip into another, but no matter which one you use, it's unlikely to look great first time. Experiment with the duration of the transition; often shorter is better, perhaps just a few frames. Also experiment with the exact timing of the transition, by rolling the edit back or forward a few frames. Sometimes this takes a little tweaking.The most reliable solution is to coach your subject, asking them to pause, reset and relax between sentences. If they can get back into roughly the same position with a similar facial expression before starting each sentence, you'll have much more luck trying to mash one shot into another than if each shot is wildly different.Still, sometimes this transition simply doesn't work. In those situations, try to mask the edit another way: cut to b-roll, cut to a different angle, punch in with a quick zoom on the second clip. The usual non-AI techniques are still available; this is just another tool in your box of tricks.Something else that could be useful is retiming, and while Optical Flow retiming is common across many apps, the best machine learning slow motion is currently found in Final Cut Pro. Let's slow down some clips.

Video Retiming

While it's still best to record in a high frame rate for the best quality slow motion effects, sometimes the decision to slow video down is made after shooting. Also, most cameras face a limit in the maximum frame rate they can offer at a particular resolution, and it may not be possible to capture at the resolution and frame rate that's desired.As a result, it may be necessary to create new intermediate frames in the post-production process, and a method known as Optical Flow is available in many apps. While this algorithm has been useful, new Machine Learning algorithms have been added in Final Cut Pro, in the Super Slow Mo feature, and they're a huge step up in quality.

Figure 4.23 – The Smooth Slo-Mo option uses machine learning to interpolate needed frames, with excellent results

With a clip selected in a timeline, choose Smooth Slo-Mo from the Retiming menu below the Viewer, then pick a percentage to use a starting point. Using advanced interpolation, new intermediate frames will be created to fill any gaps, and the AI does an excellent job. Compressor also offers this advanced retiming option.While DaVinci Resolve and Adobe Premiere can't quite match this quality just yet, free apps like FlowFrames could be a useful substitute, and it's also possible in an app better-known for upscaling: Topaz AI Video.

Upscaling images and videos

Increasing the detail or resolution in an image is a task which image processing utilities have handled for years, but without AI, there are limits as to how far you can push. Videos also sometimes need to be sharpened or upscaled, and this can be a more specialized task. Today, AI algorithms are offered in many applications, and have become somewhat mainstream.In production terms, these tools are simple to use; you'll input a low-resolution image, run the process, and you'll receive a higher resolution image. Depending on how bad the original image or video is, you may prefer to process it before you upscale, or afterwards. In general, if an image is really poor, you may wish to upscale first before doing a manual cleanup pass afterwards. However, if you're starting with a decent image and trying to make it very large, try this:

Save your original layered image.

Save a flattened copy of that image.

Upscale that flattened copy.

Why? Simply, very high resolution images with layers can become quite large and hard to work with. I've upscaled images for trade show displays that have ended up over 500MB for a single image, and while that isn't much in video terms, it's slow to load and save in Photoshop.For best results, try to exactly double an image's size. In fact, many tools will only make an image 2x larger, and if you need more than that, you'll simply run the process twice. As usual, the more you try to do, the less likely you are to get away with it — there's no substitute for real pixels, and if you want to cover a large display at 300dpi, you should start with a high resolution image.While there are many options (read on for some suggestions) probably the best known image and video enhancers come from Topaz Labs. They offer a suite of desktop products: Photo AI, a cheaper, upscaling-focused Gigapixel AI, and a more expensive Video AI. In addition to the desktop apps, Topaz have several web apps that offer upscaling and other improvements for images and video. For most production needs, I'd recommend local processing, but if you have a lower-powered machine, you may need to use cloud processing at an extra cost.While there is a generative component in these all apps — they do invent detail, after all — I've included them in this section of the book because they're not trying to create something from nothing. However, at their more extreme settings, some of the shapes and textural details produced by these apps can tend towards the artificial. To avoid this, be sure to capture at the highest quality possible and use these as a slight step up.The results from these apps can be very good, though processing can be slow. AI-based noise removal can really help clean up grainy images, and the other tools here help with recovering from missed focus, with color balance, and of course upscaling helps when you need more resolution than you could capture.As AI image improvement algorithms have become somewhat mainstream, other apps also offer AI-based tools that can perform some of these tasks, including:

Adobe Photoshop's Neural Filters, which include a Super Zoom upscaler and a Photo Restoration module, among others, plus a Generative Upscale feature.

Pixelmator Pro's Super Resolution feature, which doubles image resolution with minimal quality loss.

DaVinci Resolve's AI Super Scale, which offers 2x to 4x upscaling with sharpening and noise reduction built in.

Upscayl, which offers a free desktop app capable of 2x to 16x still image upscaling. Paid cloud upscaling is also available.

Apple's macOS 26 includes APIs for video filters which will allow apps to upscale, perform video retiming, add motion blur, reduce noise and add sharpening, even in real time on a video call.

Figure 4.24 – Photoshop includes several "Neural Filters" but Super Zoom is probably the most immediately useful

Because many Generative AI services tend to create low-resolution images, they will often also include upscaling features. Always be careful that false details haven't been invented, and start from the best source you can find.

Summary

Machine learning continues to have a big impact on many image processing pipelines. While it's helpful to be able to find an image in a massive collection, it's even more powerful to be able to find a specific moment in a video. Once you've found that shot, it's incredibly handy to be able to relight someone's face, even as they move around. You can then use AI to help you deal with client change requests, reframe for a new aspect ratio, make a video much slower, or create a much larger version of an image.While your production pipeline is unlikely to use all these techniques, they can make it easier to say yes to a client's requests, make your production jobs a little easier and faster, and they'll still leave your creativity at the heart of what you make. We're not done yet; later in the book, in Part 4 on Automation AI section, we'll discuss techniques that could change your workflow entirely.In the next chapter, we'll look at Utility AI as it relates to text. Although creative production usually revolves around images, videos or music, text is still behind scripts, instructions and many interactions with clients, so it's worth a deeper look.

Recommended resources

AI Masking in Boris FX Continuum - Mocha Matte Assist ML: https://www.youtube.com/watch?v=YbbcUpKcQGU

FlowFrames: https://nmkd.itch.io/flowframes

Apple's machine learning video effects: https://youtu.be/EbRvY6j8d7g?si=8kRskK2AcpnorXwu

Chapter 5: Utility AI with Text

Join our book community on Discord

https://packt.link/EarlyAccess

Words are the backbone of many human tasks in the creative production space: we write scripts, we gather feedback from clients, we brainstorm new ideas, we summarize, we edit. Words are also key to many computer-focused tasks in creative applications — SRT files used for captions in video, and XML files that describe an editing timeline.Not all tasks that AI can help with should be classed as Utility AI, though. If text is being created from an idea, that's Generative AI, covered in the very next chapter. And if you're using AI for extensive processing tasks that drive your workflow, that's Automation AI, and we'll be covering that later in the book. Utility AI tasks are those that advise, help or perform minor fixes, rather than doing your work for you.As LLMs like ChatGPT are based around text, they're ideal for many text processing tasks. Tedious, time consuming and uncreative text jobs are likely to be something that an AI service of some kind can help with, and you'll have many choices available. In fact, there are so many options, and they're changing so quickly, that it would be foolish to try to cover them all.Instead of focusing on specific applications, here I'll focus on the tasks. With that in mind, I'm still happy to recommend specific tools if they have unique contextual strengths, such as running on your local device or being integrated into the tools where you're likely to need them.For most general tasks, those taking a text file as input and producing text as output, feel free to try any LLM you wish. Remember that locally-run LLMs carry fewer risks in terms of privacy, though they tend to not be as powerful.The main tasks which I'll be covering in this chapter are:

Summarizing

Grammar correction

Checking and verifying

Reformatting text

Summarizing

Wouldn't it be nice if long, boring documents could be made shorter? While not all creative production jobs involve large written documents, at some point you'll inevitably encounter a wall of text that you don't actually need to read word-for-word, and that's where AI can help. Though I've classified this as Utility AI, this is a somewhat Generative task — albeit one based heavily on provided input.If a document is long, and you need to know the key points quickly, there's nothing wrong with a quick summary to get you by. However, if a client has sent you detailed instructions on exactly how to fulfil their brief, it's your job not only to read the whole document yourself, but to read between the lines to discover the best way to meet the client's needs. Don't be lazy! Take the time to read the whole thing, and you'll do a better job.This practice may seem obvious, but recent advertisements from a very well-known company featured their AI assistant summarizing a client's brief because the creative couldn't be bothered reading it. AI should never be used because you're too lazy to do your job properly, but that doesn't mean it has no use at all.AI Summaries today go far beyond long documents, and can be found integrated with email applications and operating systems. Apple Intelligence, free with macOS, can summarize notifications and emails in the built-in Mail app, while Microsoft's CoPilot handles summaries in Outlook. It's even possible to use a browser (such as Arc, Dia or Comet) that includes AI features like summarizing.

Figure 5.1 – Holding the Shift key while hovering over a link produces an AI summary of the linked page in the browser Arc

Search engines include summaries too: Google's AI tools summarize almost every Google search result, with the result that publishers have seen significant reductions in visits to their sites. Searchers are more likely to stay on Google than to follow the links to the original sources of information. (https://arstechnica.com/ai/2025/07/research-shows-google-ai-overviews-reduce-website-clicks-by-almost-half/)

Figure 5.2 – Google's own summary of its AI summarization feature

Sadly, these summaries often contain inaccuracies, famously once recommending including glue in pizza toppings (https://www.techradar.com/computing/artificial-intelligence/googles-ai-overviews-are-often-so-confidently-wrong-that-ive-lost-all-trust-in-them). Worse, when used for product research, these summaries can be profoundly misleading, recommending products that don't even exist, and failing to mention any issues with a product (https://housefresh.com/beware-of-the-google-ai-salesman/). Most people won't click through to verify a summary as fact, but you should.As is usually the case with AI, mostly correct can still be useful, as long as you can refer to the real text to verify any key points. An email summary that helps you spot a needle in a haystack is helpful, while an obviously inaccurate summary can be ignored. On the web, it's always worth clicking through to the original source(s) quoted.Let's return to summaries of long documents, as not all LLMs are equally good at this task. In a recent test, comparing summarizing functionality using technical articles, Gemini came out on top. Dan Russell (https://groups.google.com/g/searchresearch-wednesday-challenge/c/t5AJTwBPvAY) used a simple prompt to compare Gemini 2.5 Pro, ChatGPT 4o, Claude 3.7 Sonnet, Grok, Perplexity, and NotebookLM:

I am a PhD computer scientist. Please summarize this paper for me.

As most creative professionals won't be working with technical articles, these specific results aren't likely to be directly relevant, but it's worth trying a similar experiment with your own files to see how much these models can vary from one another.Models will continue to change and evolve. Also, remember that each time you run a query like this, you'll receive slightly different results. AI is not entirely predictable, and if something fails, it can be worth simply trying once more.

Prompting an LLM

Prompts are important, and while a simple prompt is often enough, you'll sometimes get better results by using complex prompts, specifying how many bullet points you want to be presented with, a maximum word count, or even asking for a summary in a specific format, such as "Introduction, Key Points, Conclusion". These strategies will change between documents and with your own preferences, and many people are happy to offer their advice on advanced prompting — don't be afraid to experiment.

Some people find it useful to explicitly define the LLM's role, its task, and the desired format in each prompt. While it's not always necessary to be this specific, if a prompt isn't producing the right results, this Role Task Format (RTF) concept will guide an LLM closer to your goals:

Role is effectively a job title for the LLM, which will define the approach and tone it uses for its analysis.

Task is the job you're giving the LLM, and if you're asking for text to be transformed, this can provide important context.

Format is the output you hope to receive.

Combining all those for a summarization task, you could end up with:

You are an analyst. Please generate a summary of the attached files that's no more than 400 words long. Format your output in Markdown with headings and use bold and italics to highlight key points.

The more specific you are with your prompts, the more likely you are to get what you want from an AI model — but not everyone agrees on the best strategy. For one example "prompt optimizer prompt", visit this YouTube video:

https://www.youtube.com/watch?v=1r5Wa5hYBEw

Meeting summaries

In an office context, an Australian Government trial of CoPilot found that senior employees were more likely to use the summarization tools, which makes sense, but almost everyone who attends meetings can benefit from summaries generated from searchable transcripts of those meetings.In terms of creative production, transcribing a client meeting will make sure you can recap any key points afterwards, but a summary of the key discussion points will help you ensure you haven't missed anything important. Imperfect is still helpful, and transcribe-then-summarize services are quickly becoming normalized in medical and mental health services. Creatives who meet clients should follow suit, because it's impossible for anyone to recall every key point from every meeting.To get the most out of a meeting summary, read it soon after the meeting concludes, and check that all the key points still fresh in your mind have been correctly extracted. If anything's missing, add it in yourself. If any hallucinations are present, remove them. It's better than nothing, but if you don't check it while it's fresh, you can't fix errors.

Book summaries

What about research? If you hope to gain new insights from a book, you're not going to find them on the dust jacket, nor in an AI summary; details are important. Of course, there's not enough time in the world to read every book. If you only have time to read one book on a subject, but must choose between several potential options, summaries might help you make a better decision as to which one(s) are worth your time.While an LLM should be able to provide a meta-analysis of what a book covers and how well it's been received, unsurprisingly, these summaries will not always be accurate. The more you know about a subject, the more likely you are to find fault with results in that field.For example, I wrote a book about Final Cut Pro in 2020. It's still in print and regularly updated, and I know the field well. Asking ChatGPT "Which are the best books to read about Final Cut Pro?" does (happily) include my book, but the first beginner recommendation is for a software version not sold in over a decade, and the Apple Pro Training series recommendation is also years out of date.

Figure 5.3 – Books about Final Cut Pro — yes, I'm biased, but there are clear flaws in this list

If an AI can't give a good answer about a subject I know well, why should I trust its answers on subjects I'm not as familiar with? Book recommendations on any other topic will sound as confident as this one did, but without specialist knowledge to confirm their accuracy, the advice could be wildly wrong. If at all possible, ask a human, in person or by reading real human opinions, because you can't trust an AI completely.

Verification is key

Summaries of content you can't access can be problematic, while of course, it's easier to verify accuracy of an email summary or a news article because you can read the email in full yourself. We are drowning in news, and any help to deal with the firehose of information is welcome, as long as it's accurate.To me, Apple's Mail and Messages summaries are good enough to be useful, and I leave them turned on. But for a time, Apple wouldn't create summaries of news articles, because soon after release, some summaries of news stories were incorrect.Because misrepresenting a news source carries more reputational risk than misrepresenting a typical email, Apple has been cautious here, deactivating the feature and upgrading it before re-release. With the risk of inaccuracy in mind, if your email provider includes a summarization tool, try it out and see if it's useful for you.If you need to synthesize the key points from long documents, any LLM should be able to help, and these results should be easily verifiable. For convenience, consider an AI browser if you do this often. As ever, confirm all details yourself, and if you can't do that, be wary. Results that look good might not be accurate.

Grammar correction

Good writing is key to good communication, and this space is crowded with AI-powered options to help you avoid mistakes. While I might like to think I'm a capable writer, the tools built into macOS and common word processing applications routinely spot errors that I haven't been able to train myself out of.Some errors flagged by a grammar checker are straightforward, while in other cases, following their guidance would remove the flavour and style from a sentence. If you want to stand out as a writer, you need to maintain your own voice, and sometimes it's best to prioritize that voice over strict technical correctness.However, if you're not yet confident enough in your own writing to disregard AI advice, feel free to take it. This advice is often worth considering, even if you ultimately decide to ignore it. For example, I want to retain the "own" in the first sentence of this paragraph, though grammar checkers flag it as redundant.Most word processors have included some sort of grammar checker for many years, including a mix of heuristic-based and machine-learning features. More modern options, such as the well-known Grammarly (https://www.grammarly.com/) are more heavily AI-based, and go further in their suggestions. Often these suggestions are helpful, while at other times they feel (at least to me) overly aggressive, trying to remove anything remotely quirky.

Figure 5.4 - Grammarly's Pro suggestion, which I chose to ignore

Help like this will be useful to a novice, but accepting all suggestions blindly can lead to bland, overly homogenized text. In creative writing, quirky can be good; not every document needs to be readable by every audience. Also somewhat problematic is a lack of context, because Grammarly sits on top of a word processor rather than truly living within it.To be fair to Grammarly, their advertising mostly targets students and office workers, who have different needs to creative professionals. If you're in creative production, the world needs more unique voices, and if I'd accepted every suggestion Grammarly made to me, it would have changed the intended meaning and tone. I'd recommend fixing errors, but be careful to maintain the flavour — sorry, flavor — of your own text.

Integrated grammar checkers

The tricky part about using AI to selectively improve your writing is that if you want it to be integrated into your writing platform of choice, you'll have to be willing to tolerate a somewhat noisier writing experience. Common errors (such as you're/your or its/it's) should of course be corrected, and Word's red squiggle has made fixing the most obvious errors easily correctable for some time.Grammarly adds many more colored underlines you may find distracting, but the tradeoff is a reduction in friction. Right-clicking an issue means you can fix it in place rather than sending the entire document off to an LLM.Grammarly's integrated approach places its fancy underlines into Word, Apple's Notes app, and any apps that use Apple's standard text services, among others.

Note that you may wish to block Grammarly from appearing in text processing apps like BBEdit, because suggestions designed for standard English-language text are not always relevant when working with code, scripting or markup files.

Another integrated option on Apple platforms, Apple Intelligence, is available in Apple's own apps, and the Proofread feature offers at least basic grammar checking. (The Rewrite feature offers more extensive changes, and will be examined in the next chapter.)

Figure 5.5 – Apple Intelligence's Proofread, as seen in Notes on an iPad

After you activate it, Proofread does a competent job of highlighting problems it plans to fix, and it's possible to approve or dismiss each one in turn. While basic spellcheck works as it always has (hooray for red squiggles) this more extensive analysis only appears on request. It's not "always on" like Grammarly, which helps if you find that approach too intrusive.Word's Spelling and Grammar check works as it has for many years now, but there is an AI assistant available too. Microsoft's Copilot is built into Word (and other Office applications) but avoid the CoPilot sidebar, as it isn't actually well integrated into its parent app. The sidebar feels more like a separate app running alongside Word rather than an integrated feature. For better integration, select the text you'd like feedback on, right-click, choose CoPilot, and then choose Writing Suggestions. It's best to select at least a few paragraphs at once, becuase this will give CoPilot more context for its feedback. To me, the suggestions here also stray a little too far into generative territory, and we'll examine those in the next chapter of the book.

External grammar checkers

Of course, if you need only a specific sentence to be checked for grammar, CoPilot would do fine, and in fact, many LLMs could be suitable if you're happy to copy and paste.

If you are considering using an LLM, make sure that private data isn't being used to train it, either by setting the appropriate preferences, or by using a local LLM instead.

So — should you go the integrated path or not?I would suggest that the end goal of using an AI-based grammar checker should be to improve your ability to write well in the first place. If you find that Grammarly helps you to do that — fantastic. But if you find its presence in your word processor intrusive, you could do far worse than Apple Intelligence's Proofread, Microsoft's CoPilot, or simply sending an entire document to an LLM and asking it for pointers. Here's an example of sending a paragraph to Claude:

Figure 5.6 – Claude's feedback on this draft paragraph is mostly sound, but it doesn't understand what I was doing with my em-dash

That way, you'll be explicitly told which sentences could be improved, clearly separating the creating and editing stages of writing, rather than ticking a "fix it for me" box as you go along. Here's the same query sent to ChatGPT:

Figure 5.7 – An earlier draft of the preceding paragraph, plus pointers on how to make it better

If you'd prefer to give more context, ChatGPT can be fed a full document, and will not only return a document with changes implemented, but will tell you exactly what you got wrong. Claude told me most of my mistakes, but didn't offer a corrected file until I asked for one.Sending a document to an LLM also scales nicely to larger production tasks. If you've been working with colleagues on a larger job, each writing your own section, an LLM will be able to take a meta-view, noting ways in which the tone or style of each separate part might differ. I'd still recommend performing final fixes by hand, but an AI's opinion could be a great way to get started.Lastly, you may wish to consider writing directly in an AI-focused writing tool, taking advantage of more advanced tools to assist with writing. As many of these tools tend towards Gen AI, we'll examine these in the next chapter.AI writing assistance is available in many forms. You may prefer to write and edit in separate passes, in which case you should experiment with different LLMs to see which one you prefer. If a more integrated approach suits you best, try Grammarly, Apple Intelligence, or a dedicated AI writing app.Personally, while I appreciate the help of all these options, I'm not ready to trust any AI to produce perfect output every time, and I do want to know what I — or my transcription software — got wrong. It's a common thread: AI is useful, but imperfect. Using one AI service to check and improve another AI service's output is a smart way to minimize mistakes.That idea, asking an AI's opinion, leads neatly into the next section.

Checking and verifying

All kinds of plans, opinions and stated facts are presented in the course of a creative job, and it's certainly useful to be able to verify them quickly. As many verification queries can be expressed as text, it's straightforward to ask an AI if a work plan has flaws, if an essay currently fails to cover any key points, or if advice you've been given regarding a computer purchase is sound.Of course, a common thread has been that AI advice is fallible, so it's going to be worth checking multiple different LLMs and following any links they cite. But if you think something's probably correct and just need a quick second opinion, an LLM could fit the bill — of course, if there are no suitable humans around.Presenting your text to an AI model is easy; you can copy and paste, upload a document, or use an integrated solution like Grammarly. To test accuracy, I've presented an LLM with:

A bullet point summary of an early draft of key points I planned to write about in this chapter, and asked for feedback

An article I recently wrote about stereoscopic video, which I asked it to check for accuracy

Advice received about which Mac might be suitable for video editing tasks, and asked it to verify this

A question about the standard amount of bleed on a print document

A question about advice for a 4K video production workflow

These are all topics for which I can offer good advice myself, which puts me in a good position to judge accuracy. Happily, the high-level advice given on all of these was sound. Notes on each in turn:

The advice given about topics to cover in this chapter was solid, but as it didn't understand the context of the structure of this book (Utility, Generative and Automation AI), its suggestions didn't neatly fit into the Utility category. Still, if you're planning to write any essay-style non-fiction content, an LLM can provide a useful sounding board to make sure you haven't missed anything obvious.

Figure 5.8 – ChatGPT had a few pointers, but without a broader context its suggestions weren't always useful

Claude was able to verify several points mentioned in the article, and was able to provide verifiable links to articles online to back many of them. These included rules about how far apart to space two cameras, specialized hardware devices and resolutions of cameras mentioned.

Figure 5.9 – Claude had positive feedback on this article, and found links to back it up

"Do I need a Mac with a Max or Ultra chip for video editing?" is a question frequently answered on social media, with nuanced answers that can be summarized as — you don't need one for simpler editing, but it'll help with complex tasks. The answers given were accurate.

Figure 5.10 – Claude knows enough about specs of current Mac models to answer this question well

This simple question is one which a digital designer moving into print might need answered, and the response is correct and comprehensive.

Figure 5.11 – How much bleed is standard in print? I say "usually 3mm, follow the specs" but this answer is comprehensive

This question proposes a bad workflow, involving using an old hard drive, and deleting the source files at the end of the job. Both Claude and ChatGPT responded correctly, saying it was risky, then proposed better alternatives. Not all the advice was perfect, but it was a great step for further research.

Figure 5.12 – This relatively complex question goes beyond what you can expect from a search engine, and the answer is more comprehensive than you'd get from a forum — very helpful

Imperfect advice

That last point is worth digging into, because workflow questions can be tricky, and not all commenters agree on the correct way forward. I followed up with further questions, including "What if I'm working with 6K multicam sources in Final Cut Pro" and then "Do I have to store the original media inside a Final Cut Pro library?" ChatGPT did very well, while Claude included some questionable (but commonly repeated) information.

Figure 5.13 – Claude's answer to a follow-up question wasn't perfect, and the deeper you dig, the more you should be talking to a human

For example, Claude stated "Final Cut's multicam clips create additional rendered files during sync and editing" while these features, optimized multicam clips and background rendering, are both optional. It also states "FCP creates optimized media and proxy files that can double your storage needs" which is technically correct, but makes the process sound like both formats are required, while usually one or the other is used.In short, the explanation was mostly correct, but had more flaws than ChatGPT's. The only major flaw in ChatGPT's output was assuming that a 6K master file is larger than a 4K ProRes 422 HQ, which isn't true if a more compressed codec is used for initial capture.So is it worth using a common chatbot for verification purposes? I'd say yes… mostly. If you know enough about a topic to phrase a question thoroughly and correctly, you should be OK. Since graphic design, video and audio questions have long been asked and answered by countless humans on countless forums, LLMs are able to synthesize correct answers to many timeless questions.Because LLMs have been trained on all those forums, as long as you keep your queries relatively high-level, an LLM is likely to deliver the collective wisdom of the internet. However, the more specific an answer you need, the less accurate an LLM is likely to be — though they do pretty well overall.For feedback on a complex document or a new workflow, LLMs certainly seem to be able to help. A human may be able to provide more specific answers, deeper insights, or an alternative approach, but the answers I've been given here have been useful and mostly accurate.A related idea, asking an LLM to teach you how to perform a creative task, also has promise. Because you're able to ask complex questions and follow them up with more detailed queries, an LLM can sometimes perform the role of a personalized tutor. Because there are overlaps with AI doing your work for you, we'll return to the idea of using AI as a personal tutor in the Automation AI with Digital Assistants and Agents part of the book.Another area which has overlaps with the automation section of this book is moving text from one format to another.

Reformatting text

It's relatively common to be given data in one format and need to transform it into another. When this is a task performed in bulk, it's automation, and we'll certainly look at that in more detail in a later chapter. But if it's a one-off bespoke task, sometimes AI can help with a job you might otherwise spend a boring hour doing, or perhaps outsource to a colleague who owes you a favor.One basic, common design task is to clean up provided text or Word documents. As you may know from hard personal experience, graphic design apps expect you to separate paragraphs with only a single return and use only one space after a full stop. While there are preset GREP search patterns in Adobe InDesign to fix these issues, and even a script (FindReplaceByList) which runs them all at once, this is also the kind of task that AI excels at, fixing common typos along the way.As a test, I gave the same short problematic Word document to three LLMs.

ChatGPT did well, getting extra points for offering to create a new Word doc for me to download.

Claude went further in its corrections, correctly changing "em dash" to "em-dash". However, it didn't offer a Word doc in return.

Gemini found most errors, but produced text without any paragraph breaks at all and didn't offer a download option.

While this is a relatively basic task, what about something more complex? One task I've been faced with recently is having to renumber images in a book chapter after removing an image. A chapter might contain many images, each with a unique number in a caption, and if an early image, say Figure 4.5, was to be removed, all the images after that point must now be renumbered. The original Word document, including images, was 6.7MB. Can AI help with this boring job?

ChatGPT correctly figured out that subsequent numbers should be reduced by one:

Figure 5.14 – This list of figures extended to Figure 4.45

Then, it did a great job, offering a corrected Word document for download:

Figure 5.15 – This is exactly what I'd want a digital assistant to offer

Claude understood the problem and proposed the correct solution, but Claude can't output a Word document, and with a free account, it couldn't process the file either.

Gemini outright refused to help, saying it couldn't edit the document:

Figure 5.16 – I was surprised at the refusal to work with a Word document, but it was an issue here

A common task when working with academic content is to change the format of citations. Because this is a fairly ordered, predictable job, you'd expect it to proceed smoothly, and with major LLMs, it does. This prompt was no trouble:

Please reformat these citations as Harvard.
Delaney, M. (2025, May 16). Google's AI Overviews are often so confidently wrong that I've lost all trust in them. TechRadar. https://www.techradar.com/computing/artificial-intelligence/googles-ai-overviews-are-often-so-confidently-wrong-that-ive-lost-all-trust-in-them
Digital Transformation Agency. (2025, February). Australian Government trial of Microsoft 365 Copilot. digital.gov.au. https://www.digital.gov.au/initiatives/copilot-trial/microsoft-365-copilot-evaluation-report-full/executive-summary-glossary
Navarro, G. (2025, July 7). Beware of the Google AI salesman and its cronies. HouseFresh. https://housefresh.com/beware-of-the-google-ai-salesman/

Here's Gemini's output, though ChatGPT's output was identical:

Figure 5.17 – Gemini nailing a citation format conversion

However, a local LLM (Gemma 3) failed completely on this task, not transforming the text at all, though it was confident of its success, apologized when told of its failure, and then failed again. A local copy of DeepSeek did better, but failed to list the day and month of publication before the words "Available at":

Figure 5.18 – A distilled, local DeepSeek not quite getting there

Note that generating these original APA-format citations from just their URLs is covered in the next chapter, and it did not proceed as smoothly as you might hope.

Here's another one. What if you've been sent a somewhat casually-written list of names and positions, and you need to generate "lower thirds" title cards for these people in a video you're producing? A short list won't take too much time, but a longer task along these lines certainly can. Just ask an LLM to clean it up for you:

I've been sent a list of names and positions in an email, listed below. Can you please extract all the names and positions, expand any contractions to the full titles, put everything in the same order, and sort the list into alphabetical order based on surname?
You'll be interviewing MD John Smith, Dave Jones who's the CTO, Jenny Davis the Accounts Manager, Admin Assistant Dave Kelly, and Jessica Keyes, a trainee.

ChatGPT gets it just right:

Figure 5.19 – This list is only five names long, but sanitizing data would work with a longer list too

Gemini was nearly there, but it put the surnames first in its final output. Still useful, and if you ask to swap the first and last names, it gets all the way there.

Most local LLM's aren't quite there

A local copy of DeepSeek can't quite get the sorting right, and doesn't separate the title from the name, but it's better than nothing. Unfortunately, asking the model to fix the error causes it to think for a full minute before making it worse:

Figure 5.20 – How did the model throw one of the names away?

Another local model from Google, gemma-3-12b, did a similarly mixed job. It was able to extract the full names and produce a list, but it got the order wrong just like DeepSeek.

Figure 5.21 – Gemma's output — useful, yet imperfect

Unfortunately, asking the model to re-sort the output didn't quite work either.

Figure 5.22 – Dave Jones is now after Jenny Davis (good) but Jessica Keyes should be after Dave Kelly

I also tried this query with the on-device AI model available in macOS 26, in an early developer beta. It failed in interesting ways too:

Figure 5.23 – Contractions not expanded, not in order, not what I wanted

Happily, the local model released by OpenAI (gpt-oss-20b) was able to perform this task flawlessly:

Figure 5.23 – The open version of ChatGPT seems to be more capable than other open models

It seems clear that although local models preserve privacy and are free, they're mostly not as powerful or reliable as the cloud-based models — though OpenAI's effort bucks the trend, and is worth downloading. For more complex tasks, if a cloud model can do the job quickly and you're able to upload the data online, that's likely to be more efficient.Although the local models couldn't quite get the order right, if you can ask for the data to be in a tabular format like CSV, it's easy to open and sort the data in a spreadsheet like Excel or Numbers. Also, if you're struggling to get good results, consider showing the model an example of what you're looking for.

Figure 5.23 – Giving an AI an example of what you want from it is an easy way to improve its output, but the on-device model doesn't know that "MD" means "Managing Director"

Export formats are key for production

There are many other text processing tasks which prove tedious for humans but which aren't too much for an AI. For example, if a client is changing the style of their documents, and while they previously used just italics to indicate quotes, they'd now like to revert to a more standard pattern with "quote marks". Can AI help? When asked, Claude thinks it can:

Figure 5.24 – Claude suggests a good solution and seems to be on the right track

While the offer is great, Claude can't actually fix a Word document and provide it for download. Copying and pasting fixes manually isn't that helpful, and while ChatGPT is capable, regularly uploading files isn't really feasible on a free account.In fact, if you ask the major LLMs what they can take as input and provide as output, only ChatGPT can actually read IDML files (InDesign Markup Language) that could be useful in a graphic design workflow. It's limited to text-based rather than binary formats, and while it sometimes offers to create IDML files which can be opened in Adobe InDesign, this seems to be a hallucination.Here's one example of a task I hoped would work:

Figure 5.25 – This tedious, non-trivial task is something which ChatGPT is happy to attempt — but you'll quickly need a paid account for file manipulation

Right now, despite the confidence, this process fails. An IDML file was indeed created, and it did open in Adobe InDesign, but it looked identical to the file I uploaded. Curiously, when ChatGPT is asked explicitly what formats it can create, IDML files are not on the list — a shame, then, that it pretended the task was possible. While this is a dead end for now, it's worth trying again in the future.To be fair, IDML is a complex, compressed format, and many formats used in production are similarly tricky for a text-based system to work with. However, you may have more luck creating simpler formats, such as the text-based FCPXML files which work with Final Cut Pro. Still, images, videos and text files aren't enough to get most jobs done, and not every file can be exported to a text-based format. Complex format support remains a serious roadblock between LLMs and creative production.However, since input of IDML is possible, and any InDesign layout can be exported to IDML, you can ask ChatGPT to work with the text in your layouts. And since many editing applications can export to XML-based formats, you've got a fair chance of extracting any text from titles or clip names in a video timeline.So far, ChatGPT has more export flexibility than the other leading LLMs, but as the field changes rapidly, I'd recommend running your own tests on your own files to see how they do. Start free, then move to paid (on a monthly basis to avoid lock-in) if you find it valuable. Do note that you'll hit the limits of a free account fairly quickly with regular use, especially if you use analysis or upload large files.

Summary

If you're working with text, you're in luck — AI can provide an assistant that's pretty good at boring text processing jobs. As long as you can avoid becoming so addicted to summaries that you skip all the details, they can save a decent amount of time. You may end up using integrated summaries in your email application more than anything else, however.Grammar correction is a task that can be integrated or separated from the writing process, depending on your preferences, and while AI does enhance this process, it should be used with care.Verification, especially when applied to common design workflows and questions, is a task for which AIs are well-suited. When you're asking about facts near to your existing knowledge, you should be able to spot obvious issues, but be careful not to stray too far from your expertise.Text transformations can be a powerful time-saver in design and video workflows. Many text processing tasks are dull, and too hard to automate by traditional means — so throw them to an AI. You just might have to start paying for the privilege.With all these tasks, be sure to check the output carefully, and never simply copy and paste. These jobs might be on the less controversial side of AI, but if you present a hallucinated summary to a client as fact, the egg is on your face.Next up, we'll stay with text, but shift gears to Generative AI.

Chapter 6: Generative AI with Text

Join our book community on Discord

https://packt.link/EarlyAccess

As we move into the Generative AI section of the book, it's worth recapping the ethical flags which any use of this tech may raise. To be clear, while not all use of Gen AI is innately bad, I would not recommend making regular use of generative text for anything you send directly to clients.If you over-use Gen AI — let's say, you decide to ask AI to write your emails for you — then the mental muscles you use to produce text will atrophy. You'll fail to develop new ones. You'll lose uniqueness in your voice, and clients will have fewer reasons to choose your services over anyone else's.If you're in creative production, creativity is not a side hustle. You need to create more interesting work than other people do, not just pump out as much sub-standard slop as possible. Every email should be a chance to practice clearer communication, not a chore to outsource.With all that said, there's definitely scope for Gen AI to create text that helps you to do your job more efficiently, or to give you ideas, or to help make your work more accessible. Let's take a deeper look at:

Ideation

Rewriting text in a different style

Generating citations

Translation

Alt text descriptions for accessibility

Not all creative professionals will need all these features, but most of us can find a use for several of them. Let's kick off with a bit of brainstorming.

Ideation

Creative jobs vary widely: some are self-directed, some are fully directed by a client's brief, and many are somewhere in-between. When you're faced with a creative decision about how to proceed, a blank page can sometimes feel daunting, and using an AI to come up with ideas could be useful.As usual, you'll get better results if you ask more specific questions, so the more you know about a subject, the better questions you'll be able to ask. Also, if you only push a little outside your comfort zone, you'll hopefully be able to spot bad advice when it's given. So, if you're a video creator wanting ideas for YouTube videos, just ask:

Figure 6.1 – All ten ideas (not all shown here) are solid

These are all good ideas which could easily be turned into a short video. While I could have come up with a similar list myself, results like this can easily spark further and better ideas. Similar prompts on different topics also give good results. What if you're a digital artist who likes dogs?

Figure 6.2 – Again, these ideas are mostly solid

If you're lacking in inspiration, using an LLM for ideas should at least get you started. While the results here aren't going to be wholly original, if a blank page is your enemy, these are a solid starting point to iterate from.Ideation isn't just about ideas for content you know well, though. LLMs can also help you stretch your wings a little further than you might be comfortable with — common when working in or with smaller teams.For example, if you're a videographer who normally covers events and you've been asked to make a promotional video, an LLM can provide you with a checklist of the typical things that a video should include. Here's the top of Gemini's response:

Figure 6.3 – The points included here (not all shown) are relatively obvious and sensible

If you're a skilled video editor but not confident with direction, this provides a starting framework you could develop into a client proposal and then a shooting script. Along similar lines, if you're a designer but not a board game marketing expert, perhaps an LLM could help you discover the best places for you to place the ads you create and niche-specific elements to include:

Figure 6.4 – This response is comprehensive and useful, with many specific points that aren't obvious outside the board game community

While I can't confirm how accurate the advice offered in other specific niches is, the above responses are in areas I know well, and they're solid. While ideation is certainly a Generative AI task, it doesn't have to take away your creativity, and could in fact inspire you further. To make sure you don't come to rely wholly on AI for inspiration, ideate on your own or with peers before you ask an AI for help. That way, you'll be getting a second opinion rather than outsourcing the job entirely.Now that you've got some ideas, hopefully you've been able to write something. What if you're not quite happy with it?

Rewriting text in a different style

Although the gap between summarizing and rewriting is narrow, I do think there's a distinct difference here: a summary is usually provided for your benefit, while rewritten text is more transformative, and usually intended to be public-facing output. The line between Utility AI and Generative AI is somewhat fuzzy here, as it was with Summarization, and ultimately, it's up to you where you draw the line.While I wouldn't regard most uses of a grammar checker as rewriting text, the same tools that offer grammar assistance (like Grammarly, CoPilot and Apple Intelligence) can absolutely give you more help if you feel you need it. Of course, general purpose LLMs like ChatGPT and Claude are happy to help with this too, as are specialist writing assistants like Spiral.

Novel-writing tools

Planning to write a novel? You might want to engage a dedicated tool to help you. Writing assistance tools like NovelCrafter, Sudowrite and Squibler are optimized for stories and novels, and as the output they produce is targeted towards more descriptive prose for fiction applications, these tools are less suited to more typical creative production text tasks like proposals, emails and pitch decks.

Here, we'll focus on more general-purpose writing assistants, and of course, this list is not comprehensive — new tools pop up all the time.

AI tools can rewrite for clarity, brevity, and in the style of a publication or author, but you can request more or less any style from most LLMs. As a test, I processed the opening section of one of my recent articles on https://provideocoalition.com, about experiments in stereoscopic video. My style is perhaps a little quirky and not entirely error-free, but that's fine by me:

Spatial 3D stereoscopic video is easy to shoot on modern iPhones, but you'll run into its limits pretty quickly. Third-party apps can improve things, but they have limitations too, and if you're used to shooting with a couple of dedicated cameras already, why not use them?
Well, if you're tempted to try making some stereoscopic 3D video with a couple of cameras sitting on the shelf, you should know what you're in for — it's harder than you think and there are plenty of mistakes to be made. On your first few stereo shoots you're bound to get something wrong, but if you practice now, you'll hopefully avoid catastrophic failure down the road when you get hold of a real camera.
Experiments like this are a great way to push your skills, but don't get your hopes too high. Even if you don't use something this janky to shoot real client projects, making it work with what you've got will give you real-world experience shooting in 3D that you can use if you step up to a pro setup down the road. So, with some regular 2D filmmaking gear to hand, how can you hack together a quick stereo 3D rig for viewing in a headset or on a 3D display?
As an extra reminder that this is an experiment, I'm not worrying about proper 3D Spatial Audio at all. Video's the focus here.

After processing this text in many different ways, across many tools, I didn't get many results I'd be truly happy to use in a professional context. Because everyone's needs and styles are different, you may have more luck than I did, and of course the tools will evolve over time.As a first pass, I started with the default "Rewrite" options. Take a look at what Grammarly suggested:

Figure 6.5 – Grammarly wants to make plenty of changes, but not all are helpful

Besides the near-complete loss of my own writing style, "limitations" are not the same as "restrictions" and "simple" does not mean the same as "janky" — I don't want a "rewrite" to change my intended meaning. CoPilot's first attempt was worse, though:

Figure 6.6 – CoPilot isn't doing a great job here

This feels robotic, lacking flow and direction. Two alternative options were no better; the second was a single paragraph long and the third was overly-salted with flowery language. How about Apple Intelligence? It didn't try to change as much as the other models:

Figure 6.7 – More of the original text was retained by Apple Intelligence's Rewrite

Apple Intelligence didn't make as many revisions to the text as the other tools, and as a result, the changes made didn't change its overall style or feel. But what if you need a more formal result? Let's stay with Apple Intelligence and ask for a Professional tone:

Figure 6.8 – Professional has a more verbose, less casual tone

While this feels a little more formal than I'd personally prefer, the meaning remains intact and the feel isn't too far removed from the original. ChatGPT's "technical" offering was OK, though it differed more from the original:

Figure 6.9 – Caption

ChatGPT offered Casual, Technical and Promotional rewrites; this was the second of those. Though there are a few points where the word substitutions have changed the meaning (including "simple" in place of "janky") it's mostly sound. Here's Grammarly's "formal" offering:

Figure 6.10 – Grammarly, offering a "Formal" rewrite

While this is certainly formal, all the flowery adjectives don't make it clearer. I think CoPilot did a better job with its formal attempt, though I did have to request "formal tone" manually, while the other tools discussed so far offered this option by default.

Figure 6.11 – CoPilot, asked for a "formal tone"

While examining these examples should be useful as a starting point for assessing these tools' capabilities, they will differ every time you run them. If you don't get the result you were hoping for, simply try again, or vary your prompt slightly. Perhaps as a way to invite experimentation, ChatGPT proactively offered a variety of styles (Casual, Technical, Promotional) and then offered to combine all three into a single "condensed best of" version. Hilariously, this rewriting of rewriting mangled the "if you're used to shooting with a couple of dedicated cameras already" fragment into "if you already have a pair"… which carries a distinctly different connotation:

Figure 6.12 – The condensed "best-of" version that ChatGPT offered — not quite what I meant

Even though I probably won't be using these tools much myself, if you're still finding your feet as a writer, I can see the appeal. However, I'd recommend starting with Apple Intelligence (using Notes or Pages on a modern Apple device) because it maintains more of your original work. In a world where mostly competent AI writing is commonplace, mostly competent is no longer good enough. If you plan to use these services, don't just accept bulk changes. Examine the suggestions, and implement them yourself if you agree.If you're in a position where you feel you need AI's help to revise or generate text, go beyond the presets and ask a prompt-based system to help you in a more specific way. There's nothing stopping you providing a document (or several, or your whole website) and asking an AI to imitate that style. You might also have luck asking an AI to rewrite for brevity, when you're beyond a strict word limit. While I'd recommend doing it yourself, if that's simply not an option, at least avoid the presets.Conversely, what if you haven't written the text yet, and you'd like an AI to generate whole paragraphs from bullet points? This is, indeed, possible — but is it a good idea?

Writing full text from bullet points

If something is easy to do, it has little value. A design built on a template doesn't stand out when viewed alongside other work made with templates, and AI-written text won't stand out either. In a few short points, here are my key recommendations:

Writing something yourself helps you grow as a writer

If you outsource a job you don't get better at it

AI is quick, cheap outsourcing

AI work is competent but not great

If AI starts something, you should edit it to add your own touches

If I was feeling lazy, I could just get AI to turn those bullets into text, right? Let's see if that hurts or helps my argument:

Figure 6.13 – ChatGPT — competent and well said, but that's not my writing

This text is fine. Not exceptional, just fine. But if you want great results — and you should always aim for great — I can't recommend generating text you expect clients or customers to read. If it's worth reading, it's worth writing yourself.As you might expect, spending time writing something yourself produces richer text than any outsourced shortcut. Yes, you'll need to invest time in it, but the paybacks are often worth it. Writing full-length text helps you understand the concepts more deeply, and you'll be able to stand behind those points if questioned.Growing as a writer is an undervalued benefit of writing, and it's a valuable skill. Outsourcing gets you to the end goal quickly, but that end goal is all you get. If you spend time repeatedly writing prompts, you just get good at writing prompts.Still, I understand: if you don't feel you have the skills to do a good job of writing longer text, using an AI is a tempting option. If that temptation is too strong, or you're simply tasked with too much work and not enough budget to do it properly, at the very least, you should edit the generated text, as much as you can, to make it yours. Reshape, reimagine, delete at will. The less you contribute to your own outputs, the less you'll grow, and the less value your work will have.Further, I also understand that there are vast swathes of text out there which aren't terribly important — and hopefully they're not your job as a creative professional. Legally important text should be left to the lawyers, but if it's your job to write persuasive copy, you should do your job. If you outsource your core skills, why wouldn't a client hire someone cheaper using the same AI tools as you?Finally, some services, like Spiral, promise to learn your style, and then create more work in an imitation of that style, on your behalf. While this approach does address issues around a personal voice and tone, the issue of not actually doing the work remains. You'll still atrophy creatively, you'll fail to fully understand a brief if you never engage with and further develop the key bullet point concepts, and your writing won't change and grow.If you have the option, do better work, not simply more work.Are there generative tasks which simply alleviate the drudgery? Definitely. For instance, if you need to cite your sources, there are many different ways to do so, and it's a largely technical task. Can AI handle this?

Generating Citations

Though citations are more commonly required in academia, some clients may expect correctly formatted citations in proposals, work plans or annual reports. If you're in charge of the design of these longer documents and need to create citations, you may be tempted to ask AI to help with the quite specific formatting requirements of the chosen citation standard. As a test, I asked three major LLMs to turn a list of URLs into citations, and was surprised at how variable the results were. Here's the prompt:

Turn these URLs into citations, APA style.
https://www.techradar.com/computing/artificial-intelligence/googles-ai-overviews-are-often-so-confidently-wrong-that-ive-lost-all-trust-in-them
https://www.digital.gov.au/initiatives/copilot-trial/microsoft-365-copilot-evaluation-report-full/executive-summary-glossary
https://housefresh.com/beware-of-the-google-ai-salesman/

ChatGPT was confidently wrong on at least two of the three URLs:

Figure 6.14 – The author and date for the first and third references here are incorrect

Faults persisted across multiple attempts. Gemini did a lot better with the same prompt:

Figure 6.15 – The end result, after a long "thinking" process

However, another attempt with the same prompt gave entirely different results.

Figure 6.16 – The same engine also produced… this?

I don't want to spend too much space on examples, but Claude also failed at this task, and the unreliability of all these services, on what I expected to be a relatively simple task, causes me concern. If this is something you need to do regularly, I'd recommend using a manual citation generator like the one Grammarly offers. (https://www.grammarly.com/citations):

Figure 6.17 – This manually assembled citation took more copying and pasting, but the automated options weren't reliable

While I'd like to think that this is a task which at least some AI tools will be able to perform reliably, if none of the biggest LLMs can get it right yet, I can't recommend trusting any other tools to do this correctly either. The rule remains — be very cautious sharing AI-produced output with the world unless you've checked it.OK, ideas done, document written, citations complete, and now your client wants to deliver in a different language. Can AI help?

Translation

Although translation is a task which AI tools are good at, it's not one they are perfect at. If you need to translate text that's in another language for your own understanding, AI is probably fine. If you can accept imperfect results, you'll be able to grasp the meaning of text in other languages that would otherwise remain inaccessible — and that's a great thing.However, if you need to translate creative content in your own language to deliver to native speakers in another language, do not expect that AI will be able to produce perfect results on its own.Just as with transcription, the output will be fast, and mostly correct. But while mistakes in a transcription can be corrected relatively easily, if you can't understand the language you're translating to, you'll struggle to confirm the accuracy of a translation.Website listings and video advertisements for most of the cheapest products bear this out. Where there's no budget to translate a product description into proper English, automatic translation tools are used and the output remains unedited. As a result, expectations and perceived worth are lowered even further.

Figure 6.18 – This cheap smartwatch on AliExpress has a description riddled with errors

This isn't new, of course, but if other languages can't be translated to English without fault, why should we expect English to be able to be translated to other languages? I've been responsible for designing board game manuals in English as well as in several other languages, and in each case, the translation was handled by a native speaker, laid out by me, and then checked by that same translator before being checked by others.Translators can definitely use AI translation tools as a starting point, but the subtleties of different languages and dialects need a human touch. The finer points of translation are not easy to navigate, even for a native speaker. For example, the differences between European Spanish and Mexican Spanish are subtle and numerous, and even a human-supervised translation will contain errors that require many proofreaders. Subtle translation problems are also more likely to occur in creative contexts, such as video and board games, than fact-based news articles.In terms of tools, the Google Translate website has worked well for several years, and on Apple platforms, the Safari browser includes translation features that allow you to translate any website you visit:

Figure 6.19 – Safari (and other browsers) can translate foreign-language websites

Translation apps from Google, Apple and others allow live translation of audio into text (or more audio) in another language in real time, which is helpful enough to make visiting a foreign country easier, but not good enough for creative production purposes.However, AI has distinct uses for anyone engaged in creating accessible PDFs or websites.

Alt text descriptions for Accessibility

Alt text is a text equivalent of an image, describing it for anyone who can't see the image. While alt text is intended for use by blind or vision impaired readers, it's also important for search engines. In fact, text descriptions of images are a large part of how LLMs can interpret what's in an image in the first place, using it as part of their training data.As a result, many LLMs can interpret what an image contains, and they can therefore generate an alt text description which designers can paste into apps like Adobe InDesign or Bridge, or into a website CMS like Wordpress. Many governments around the world require public-facing documents to be accessible, so this is not an optional task.While it's not difficult to write a short sentence describing a single image, describing many images can be a time-consuming process. Also, because alt text descriptions should use concise, simple language, their creation is a task well suited for an LLM with human oversight... right?Before we assess the output, here are a few best practices tips for alt text, from Harvard's Digital Accessibility guide (https://accessibility.huit.harvard.edu/describe-content-images):

Keep it short, usually 1-2 sentences. Don't overthink it.

Consider key elements of why you chose this image, instead of describing every little detail.

No need to say "image of" or "picture of."

But, do say if it's a logo, illustration, painting, or cartoon.

Don't duplicate text that's adjacent in the document or website.

End the alt text sentence with a period.

There are many subtleties to generating alt text, but as well as the above points, the alt text must include the context in which an image is presented. A photograph of a university campus used in a discussion of architecture may be given different alt text than the same image used in a student prospectus. More tips can be found here (https://www.visionaustralia.org/business-consulting/digital-access/blog/five-tips-for-writing-alt-text).The easiest way to use AI to generate alt text is to use one of the many free tools designed explicitly to do the job, including https://alttext.ai, https://ahrefs.com/writing-tools/img-alt-text-generator, or https://www.tailwindapp.com/marketing/tools/image-alt-text-generator to name just three.Alternatively, using a general-purpose LLM like ChatGPT, Gemini or Claude brings the added benefit of being able to tweak the output — perhaps by priming them with tips for generating better alt text, or by providing the context in which those images are presented. Using one of my own images as a source, here's a concise alt text description from Tailwind:

Figure 6.20 – This text is concise, but without context it doesn't tell the whole story

That description is about the right length, and gives sufficient meaning but no context. Not all descriptions were accurate, though. Ahrefs offered three options, but all of them interpreted Apple Park as a walkway:

Figure 6.21 – Ahrefs provided options, but none were correct

While a human could edit any of these descriptions to include context, a more advanced LLM can do it for you. Here, ChatGPT offers a generic description, then adds context on request:

Figure 6.22 – Providing context improves the alt text — although it's probably a little too long

So while AI-generated alt text is impressive, it is, as usual, just a starting point. Human input is often necessary, either to correct the generated alt text directly, or by asking an LLM to add new information.

Figure 6.23 – This free mobile app can describe any photo in detail — but it's not instant

Blind users already have access to free tools like Be My Eyes, a mobile app that can describe anything a phone camera can see, and it's also available for smart glasses like those from Meta AI. Automatic generation of alt text is now the bare minimum, and if you're providing alt text, you should be able to do better.We've covered a lot of ground, so let's wrap this up.

Summary

AI-generated text has potential to make many tasks easier, and in projects with formal, specific output requirements, it can shine. However, in creative fields, there are real risks in fully embracing it.If the point of your creative writing is to stand out from the crowd, you can't use the same tools and techniques as everyone else. And if you want to grow as a creative person, you can't simply outsource creative tasks.That said, there's nothing wrong with accepting help, and an AI is certainly able to give you feedback on your work as well as ideas. Ideation from an LLM has few downsides, as you're in control of what you do with its advice. Creatives wanting to stretch professionally could do much worse than ask for help from an LLM, though the closer you are to a subject, the better.If at all possible, use text generation as a tool to help you, not replace you.With that basic principle in mind, let's dive into perhaps the most controversial application of AI for creative people: generating images.

Additional resources

https://www.boia.org/blog/be-careful-when-using-ai-for-alternative-text