Enterprise AI Productivity: Success Beyond the Hype

Selim Mimaroglu

A NOTE FOR EARLY RELEASE READERS

With Early Release eBooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this title, please reach out to Pearson at PearsonITAcademics@pearson.com

Contents

Preface

Acknowledgments

About the Author

Chapter 1: Why AI?

Chapter 2: What Is an AI-Fueled Business?

Chapter 3: How to Structure the AI Organization

Chapter 4: AI Innovation

Chapter 5: AI Product Management

Chapter 6: AI Research

Chapter 7: AI Deployment

Chapter 8: AI Production

Preface [This content is currently in development.]

This content is currently in development.

Acknowledgments [This content is currently in development.]

This content is currently in development.

About the Author [This content is currently in development.]

This content is currently in development.

Chapter 1: Why AI? [This content is currently in development.]

This content is currently in development.

Chapter 2. What Is an AI-Fueled Business?

Let’s start this chapter by describing what it means to be an AI-fueled business. Listing some well-known AI-fueled companies may help you build an intuition: Google, Netflix, Amazon, Tesla, Uber, Airbnb, and Oracle. Most of these companies are technology companies, but some are not. Some are built on AI to run their businesses. AI-fueled businesses use AI at their core, which powers their business strategy, innovation, operational efficiency, products, and services. An AI-fueled business thinks natively in terms of AI:

• Its organizational structure is built for utilizing AI (a main topic of this book)

• Its executive management understands AI and supports it

• Its management is very experienced in AI

• Its individual contributors are experts in AI

These companies offer hundreds of products and features powered by AI models. For example, it is estimated that Google has developed 800 products since its inception, and each product has well over 10 features. To an AI-fueled business, training AI models and putting them in production is second nature.

There are many benefits to being an AI-fueled business. First and foremost, AI grows your business by opening new doors, enabling new products, and creating new market segments. An AI-fueled business is best situated for growth as underlying AI technologies help solve previously unsolvable problems.

Think about AI products that have changed our lives; they are considered transformational innovations—arguably the most significant type of innovation. Google Maps is a transformative product from Google that uses AI to provide predictive traffic and live views. Netflix uses AI to ensure smooth, high-quality video playback for millions of users worldwide. Algorithms constantly monitor network speed and dynamically adjust video resolution in real-time to minimize buffering and reduce data usage. Amazon uses AI-powered robotics and tools to efficiently manage its fulfillment centers. Tesla has been utilizing AI, and especially deep learning, for autonomous driving. Uber has been leading AI forecasting to predict supply and demand and adjust pricing dynamically. Airbnb has been using AI for smart pricing and to enhance the user experience with image-optimized photo tours. And Oracle has been embedding AI in Fusion Cloud applications.

Most experts believe that transformational innovation is easier with AI today than without it. Transformational innovation refers to the development of groundbreaking ideas, technologies, or methodologies that fundamentally alter existing markets. Transformational innovation driven by AI is not easy to come by, but it certainly makes the case for investing and structuring your organization as an AI-fueled business. To secure this large investment, you must show that AI contributes significantly to your organization’s total revenue.

The second-largest category of innovation after transformational innovation is incremental innovation, which improves products and features by making them better, faster, and cheaper. An AI-fueled business has incremental innovation in its DNA, which focuses not only outward but also inward. You can transform your business by making complex and tedious internal tasks and processes easier with AI. This is called operational efficiency. By improving operational efficiency, you help develop products more efficiently. Incremental innovation is the largest bucket of innovation; it is easier to explain and receive support.

Remember, all companies and products die, and companies that don’t innovate die sooner than expected. With transformational innovation, it is clear that an AI-fueled business grows by expanding into new market segments. With incremental innovation, the expansion is not guaranteed; the core of your offering is the same and may help expand your market somewhat, but not necessarily in meaningful ways. In other words, you may even lose customers with incremental innovation, but the rate of demise is slower.

The third type of innovation lies between transformational and incremental innovation. Some refer to it as sustainable innovation, not to be confused with the term that refers only to environmental and social innovation. It is sustainable because your business will remain roughly the same size; it doesn’t grow, but it doesn’t die off either.

AI-fueled businesses have executive managers with formal training in and extensive experience with AI. They have large AI teams and invest in AI education and training. They have many products that rely on AI models, and they have well-defined AI processes that are documented and shared company-wide. They possess the technology that enables AI development, and in some cases, they make their technology available externally. PyTorch, TensorFlow, and OCI Data Science are just a few of the commonly used development tools and libraries.

An AI-fueled business model comprises multiple building blocks, including people, product, process, technology, and culture, as shown in Figure 2-1. This chapter focuses on each component, highlighting the most important aspects of each tenant. If you are interested in more details, I suggest you read All-in on AI by Tom Davenport and Nittin Mittal.¹

Figure 2-1 Building blocks of an AI-fueled business model

¹ Davenport, Thomas H., and Nitin Mittal. All-in on AI: How Smart Companies Win Big with Artificial Intelligence. Harvard Business Press, 2023.

People

Individual contributors at varying levels and managers at all levels of seniority make up the people element. People at your organization form teams, collaborate, celebrate successes, undergo change, and sometimes fail. Among all the building blocks that I share in this chapter, the people element is the most challenging to get right.

Becoming an AI-fueled business is your goal, but to get there, you need a good strategy. Having a strong strategy doesn’t guarantee success, but its absence pretty much guarantees failure. Support of your people is a must-have to turn an organization into an AI-fueled business. People can make or break this goal, but they are likely to support it if they feel included in the journey and if it is accompanied by quick wins and long-term goals. A quick win can be any short-term success, including operational efficiency improvements enabled by AI, or a new product or feature backed by AI. Be wary of having only long-term goals. Remember that people are impatient, and you must have shorter-term milestones and quick wins within your long-term plan.

When developing a strategy, it’s essential to consider how you position AI. In today’s world, AI is a necessity—a powerful tool for survival. Therefore, it would be a mistake to position AI as a luxury item accessible only to trillion-dollar companies.

Investments

An investment in people is needed to reap the many benefits of AI, and education is required to correctly utilize AI in an AI-fueled business. A continuous learning mindset is a must for successful individuals and organizations that can shape their future accordingly. Education takes many shapes and forms. General-purpose education targets large audiences by providing easy-to-grasp, high-level, practical AI information. Thousands of employees enroll in general-purpose education, which helps form a general mindset toward AI. With this mindset, people are more receptive to AI's benefits and better understand potential risks and how to address them.

Note

Subscriptions to books, magazines, live-learning events, and conferences provide both general-purpose and specialized AI education. O’Reilly Learning is a good example, offering live-learning events, recorded videos, and tons of high-quality text in the form of blogs, articles, and great books from well-known publishers such as O’Reilly, Pearson, Packt, Manning, Addison Wesley, Elsevier, MIT, Harvard Business Review, and others. An organizational subscription to O’Reilly Learning brings all the benefits to your organization.

There are many other online educational resources, such as Coursera and Udemy. These providers serve a general-purpose audience that doesn’t yet need a specialty in AI, and most of the courses are designed for beginners. Many well-known universities in the US and all over the world provide certificate programs. These programs offer significant value. Certificate programs cover a wide range of needs, with some being general-purpose, while others are specialized in technical topics, product management, and C-level interests. Certificate programs can cost a couple of thousand to tens of thousands of US dollars, depending on the university and the certificate. Coursera, Udemy, and O’Reilly Learning are priced reasonably, and most organizations cover their cost with an enterprise membership. However, university certificate programs are significantly more expensive, and only a small portion of their expense is covered by large tech companies. Typically, around $5,000 (or perhaps $10,000) of the tuition may be reimbursed after successful completion. There are some restrictions to tuition reimbursement programs: most cover full-time employees only for job-related courses and certification only, with managerial pre-approval. A reimbursement program pays you back after several months of enrollment, generally after successful completion.

Some large technology companies produce their own educational content: Oracle University, Google Learning, Microsoft Learn, AWS Training, and IBM Skills Network are just a few examples. If done correctly, in-house educational content can be very useful; the bar should be high and should not compete with existing books and resources. The focus should be on value-added education not covered by mainstream learning facilities. At the end of each large learning module, an employee may receive a certification. These resources are free to employees, and most are also available to the general public, either free of charge or for a small fee.

Now that you know more about resources, let’s talk about how learning takes place and how to improve learning activities. In-house learning (for example, Oracle University for Oracle employees) assigned by management is completed during work hours; it is part of your job. Online learning from third-party resources such as O’Reilly, Coursera, or Udemy is rarely considered part of work, and in general, most should be completed before or after work. University certificate programs are very demanding and can take around 20 hours per week. These programs consist of regular assignments that require substantial research and writing and are conducted in groups. Similarly, certificate programs are not seen as part of the job but rather take place outside work hours.

Of course, in addition to all these resources, there are conference and journal publications from IEEE, ACM, Elsevier, Springer, and others. A typical AI paper might take a couple of days to read and fully comprehend. In most cases, journal content is dense and technical, and the target audience is AI researchers and engineers. Reading a technical paper might be part of your work, depending on your title and job description.

Embracing Failure

You may have heard of the concept of embracing failure within an AI context, but what does it really mean? Nobody wants to fail, but when a project fails, there are a couple of ways to handle it: the conventional (wrong) way is to keep quiet and sweep the failure under the rug. A better way is to perform a post-mortem to analyze the details and learn from the failure as much as you can. By doing so, you build experience and learn from failures, both of which are significant milestones. It is important to be vocal about failures—for example, you can share a compelling story with your business about why you think the project failed. Being vocal about failures builds trust and makes the work more real. Some inexperienced managers talk only about successes; everything in their world is pink but also fake.

Unlike conventional software projects, which have their own risk of failure, AI projects are riskier, and their outcomes are less certain. AI model architectures that work for others are not guaranteed to work on your data; the only way to know is through experimentation. Embracing failures is a significant mindset shift from classical software development, which is often lacking in most businesses and organizations. By embracing this principle, organizations achieve greater success. AI is based on experiments, and more experiments bring more success—you need only a handful of successful experiments out of tens of experiments.

The failure rate is higher on new projects for your organization. Most organizations don’t have the luxury of stopping work for learning, which means you will learn as you develop. Imagine that you are taking on a large GenAI project, and more specifically, you are fine-tuning an industry model for your line of business. You need to learn how to tackle your project from published papers and recent books on this subject. You must identify datasets for training, some of which might be owned by your business. You need to consider multiple foundational models, as there are many ways to train, many hyperparameters, and multiple metrics for validating how well a fine-tuned model performs. Depending on your project timeline and available resources, you need to plan how many experiments you can conduct. Hardware availability may also be a limiting factor, as training on large foundational models requires new and powerful GPUs. Some of your experiments will have poor results, and some may not even be concluded. Each experiment will take time and resources, but every experiment is valuable. You must be in the mindset of learning from mistakes. It takes careful examination, brainstorming, and positive encouragement to study failed experiments so that follow-up experiments perform better. Unfortunately, there are no rules set in stone; what worked for other teams or companies might not work for you. You can also learn from other people’s mistakes, if they are properly documented and accessible (usually they’re not), to steer yourself in the right direction.

Change Management

In your AI journey, you will encounter many people who doubt AI efforts. They may express their skepticism openly or convey their point of view through various means. Doubters and naysayers may be at every level of your organization: at executive, high-level, and mid-level management. Skepticism might also come from individual contributors, engineers, product managers, salespeople, or implementation engineers. You might think some are annoying to deal with (and you might be right), but it is a blessing to have open criticism. Raised concerns give you a chance to listen carefully, understand, and provide a thoughtful, well-constructed reply. Taking other people’s opinions into account is part of change management.

Change management is a structured approach for transitioning individuals and teams. Change management can be effectively used for transitioning an organization into an AI-fueled business. It may be implemented exclusively by a dedicated team or handled collectively by multiple teams and individuals. Especially in large organizations, people want to be heard, and they also want to be informed about a master plan. At regular intervals—weekly, biweekly, or monthly—meetings should be held to provide updates on what has happened since the last meeting, highlight achieved milestones, and outline upcoming targets.

Who participates in change management? You can’t expect AI personnel to take on change management solely, but they should actively participate. At a minimum, they should give presentations, share model accuracies, and hold workshop sessions. If you have a project management office (PMO) within your organization, it should be an integral part of change management efforts as well.

Hiring

Perhaps the biggest expense in the people category is hiring and retaining AI talent. AI experts are not easy to find and come at a premium. If your organization is serious about becoming an AI-fueled business but lacks sufficient AI talent, you might repurpose some existing personnel; however, you also must hire AI professionals with experience in the relevant domain.

Product

AI-fueled businesses are recognized for their distinctive products, which are widely used by many businesses and people. AI-fueled businesses develop AI-first products. In their arsenal, AI is the most powerful tool for solving problems. One common attribute of AI-first products is that they provide answers to hard problems that were previously unsolvable. AI-first products target customer pains. By pursuing transformational innovation, they develop game-changing products that deliver significant value.

The whole point of being an AI-fueled business is to deliver exceptional products and offerings powered by AI. AI is not available only to trillion-dollar technology companies; the field of AI is level, and accessing and implementing AI is within reach for all businesses across the world. Once you set up the organizational structure correctly by following the guidelines in this book, benefiting from AI is straightforward.

Businesses ignoring AI face a double whammy: first, their existing offerings are outdated at an unprecedented speed; second, competitors using AI can enter their market faster than ever before.

Data

One of the most important requirements for becoming an AI-fueled business is acquiring data. There are multiple sources of data, internal and external. Internal data enables AI model building, and an AI-fueled business continuously collects data ethically. They collect data directly from their customers and from their products. Internal data is the biggest differentiator because it is available only to your organization. For that reason, you must make investments and put in place mechanisms to collect data and make it available to your business. Benefiting from internal data sources accelerates your journey toward becoming an AI-fueled business that offers breakthrough products. In addition to continuous data, you will also collect and receive one-off data from various sources. Arguably, one-off data is less useful than continuous data, but it can still provide valuable insights. Imagine that you are building an AI model to predict enrollment in a program. One-off data showing enrollment in this program may be very useful for AI model validation, but perhaps not very useful for model training, especially if it is of limited size.

External data procurement choices are plentiful. The most common stages of data procurement are discovery, exploration, and contracting. In my career, I have led multiple data procurements. The following are some high-level takeaways that might come in handy for planning:

• Discovery is a never-ending process.

• You must search for new data continuously.

• Exploring sample data tells a one-sided story.

• You always end up with less impressive data than anticipated.

• Contracting takes forever.

Exploring data companies and navigating the contracting process is a time-consuming but rewarding task. This task may be led by a technical manager at your organization, and it must include representatives from AI who will perform exploratory data analysis on the sample data and verify that the data dictionary satisfies AI modeling needs. Note that third-party data providers are available to all your competitors as well, so they are not really differentiators.

Ethics Committee

An ethics committee guides what can be done with data and corresponding AI models. Once an AI model is built, it undergoes a review process to make sure that it is compliant. A detailed breakdown of results provides a clear picture of an AI model’s shortcomings. In most cases, a fix includes retraining, followed by re-evaluation. Large technology companies have established processes for evaluating both data and AI models. Small and medium-sized companies new to AI should establish an ethics committee if they don’t already have one.

An ethics committee can look into the following attributes: fairness, privacy preservation, explainability, robustness, and transparency. A model is not fair if it is biased for a group. In most cases, bias is an artifact of the underlying training data, which might be corrected by better data or resampling. Privacy preservation is the idea of safeguarding an individual’s privacy by avoiding unauthorized disclosures. Explainability refers to the ability to understand how an AI model reaches a prediction. Robustness refers to an AI model’s ability to produce reliable output in the presence of noisy data. Transparency has many aspects, but at its core, it involves information about training data, validation data, and accuracy metrics.

Monitor Products

You have developed an awesome AI model and deployed it to production. Now, your model is empowering products and insights. This is a feel-good moment that requires a team celebration, but you’re not done. You must monitor products to see how your customers use them. You might need to intervene if something is wrong. Monitoring is performed to detect model drift or data drift. With monitoring, you can also discover some unintended use cases about your product, which could result in new product ideas.

It is common to have a one-to-many relationship between AI models and products. In practice, an AI model often empowers multiple products. This turns out to be a good idea and helps you justify return on investment. Even though AI modeling investments are real expenses, they pay off.

Process

The way you perform activities within your organization contributes to the overall success of AI projects. AI-fueled businesses employ efficient processes to control risk and streamline work to their advantage. These processes span multiple teams, and they may start by identifying a problem or an opportunity. A description of the problem is needed to understand it well: a market requirement document, followed by a product requirement document, provides insights into the problem. Is the problem a good fit for AI, or can it be solved by another method? These questions are part of the well-known Stage Gate. The Stage Gate process directly applies to AI model development. It starts by questioning the value of a product, explores the feasibility of AI models, and then guides AI model development, deployment, and monitoring. Following the Stage Gate process aims to reduce risk. A gate follows each stage, hence the name. At each gate, a decision is made to proceed or halt, which seeks to cut losses when needed instead of proceeding all the way with a subpar product.

The Agile code development process is well-suited for AI code development. Let’s review some of the building blocks of Agile: customer collaboration, iterative development, value-driven delivery, welcoming change, test-driven deployment, and continuous improvement. Each sprint is an opportunity to focus on a feature. The Herculean task of AI model development is divided into manageable subtasks described in tickets. AI model development contains software development, which is accompanied by Agile principles.

For new products, large technology companies follow an architectural review process that can take several weeks and require significant effort. For a new product, the architectural review process involves evaluating the software architecture to ensure that it meets technical, security, and quality standards. If you don’t have such a system in place, it is a good idea to form a group of software engineers to review architectural diagrams of new products.

Each dataset, including even publicly available ones, goes through a review process. Don’t assume publicly available datasets are good for commercial use; some might be, but most aren’t. Similarly, each software development library, including publicly available ones, goes through a review process. This involves vetting both data and software libraries for development purposes. On top of all these, when you finish developing an AI model, it goes through a review process before you can use it in a product.

There may be other processes, depending on what you want to achieve, but I’ve covered all the major ones. The point is that, in an AI-fueled business, processes serve to deliver better products and services.

Overcoming AI Product Deployment Hurdles

Overall, you can think of processes as linear, with one process following another, which can take time and slow down AI development. This section discusses how to handle some of these processes in parallel to achieve speed.

In most cases, AI deployment can begin before the AI model development activities are fully completed. It is common practice to share a “good enough” AI model with the AI deployment team while continuing to work on AI model development. By sharing a model, you share both the inputs and the outputs of the model. Similarly, the model's performance can be tested to produce a good estimate. Note that some things may change once an AI model is released in production, but in most cases, intermediary models provide very good guidelines.

It is essential to verify that AI models scale effectively by leveraging multiple CPUs and GPUs in parallel. You must perform some benchmarking to identify potential problems before deploying a model in production.

Any single problem in product deployment will delay the overall progress. Familiarity with the processes, starting each process in a timely manner, and experience with deploying AI models into production help, but don’t guarantee problem-free development. In an AI-fueled business, the AI model development processes become second nature, and red tape is reduced, with only the necessary processes retained.

Commitment

The journey to becoming an AI-fueled business is an important decision that requires support at the highest level of your organization. Throughout the lifetime of this transition, you also need active participation from an executive at your organization, such as a CAIO, CDO, CIO, or CTO. Executive-level management support is required, but it is not enough; support from managers and individual contributors is also needed. Note that you need people who are willing to take action; managers and individuals who give lip service to the vision but lack genuine belief are detrimental to the overall progress.

Communication

Good communication achieves wonders. It keeps you aligned with your true objectives and wins hearts. How to enable correct communication is a big question that needs careful thought and planning. Internal and external communication are handled separately because the goals and audience are different in each case.

Internal communication is part of change management; the target audience is your peers and employees. The aim is to set overall, long-term goals while also delivering short-term wins and achievements. Internal communication is best handled by regularly scheduled live meetings, with abundant time for questions and answers. Live communication is essential, as it shares progress and addresses challenges along the way to becoming an AI-fueled business. It is important not to make this overly rosy; sharing wins is a good idea, but hiding challenges creates all sorts of problems. Live meetings should be accompanied by internal documentation with more detail, project plans, and tickets, so that interested parties can refer to raised questions and concerns in writing. Instead of live meetings, some organizations might prefer email updates. Email communication doesn’t work well for large initiatives, although it may save time. An email is a dry form of communication that is perceived as one-way and must be shorter than a live meeting.

External communication is as important as internal communication. You don’t need to wait to reach the highest level of an AI-fueled business to share your activities and points of view. One common mistake organizations make is sharing very little. AI-fueled businesses communicate externally frequently, sharing updates on their projects that are not yet products, as well as producing a wealth of useful content for their potential customers, developers, and AI researchers. External communication has a special value in business and is also known as research push. Most organizations perform only market pull; they utilize communication only for selling their products. Potential customers, individuals, and organizations are not moved by marketing and sales talks; they need evidence and a good story around the products. These are provided by research push in the form of blogs, articles, conferences, talks, and presentations. Research push combined with market pull creates a great synergy.

Education

We previously discussed forms of AI education. In this section, I share the process for education. It is a good idea to combine mandatory and optional training; completion rates for optional training provide useful data points. Collecting feedback from participants is a good way to navigate education effectively. If you are providing in-house educational content, you must compare it with others to ensure that both the style and the content are competitive. Without any competition, both style and content may suffer; worse, the education you require might not be useful for your employees.

Most specialized education for AI researchers is available in technical books, live training sessions, and recorded videos. Some companies have access to digital libraries, such as the O’Reilly Library. This is very beneficial overall for being an AI-fueled business, but the time spent on these resources is generally not considered work time. A good process should allocate a reasonable amount of work time for self-learning.

Many universities offer professional educational certificates. These programs are very useful, but they can be demanding. Similar to specialized education, they are not considered part of your job, meaning that you need to complete learning activities and long assignments before or after work hours. Similarly, a good process may include some time spent in these programs as part of work hours. There are some other constraints, especially if your company is providing tuition reimbursement. One major restriction is pre-approval from your manager that confirms the education is related to your job. There may be other restrictions regarding grades and the amounts of reimbursement. Some of the certificate programs from well-known universities could cost tens of thousands of dollars.

Technology

As an AI-fueled business, you are tasked with developing great products powered by AI. You need sufficient technology and good sources of production data to build AI models. There are many AI modeling architectures to choose from: conventional techniques, ensemble methods, modern techniques, and GenAI.

No matter who you are, and no matter what company you work for, there will always be others with better AI modeling architectures than yours. Similarly, other companies will have a better AI ecosystem than yours. Accept this fact and realize what it means to your organization—probably not much. Your key differentiator is data, not AI modeling architectures. Take a moment to digest this important information.

Although some AI modeling architectures might be better, all companies have access to similar fundamental models. Open-source AI modeling architectures are highly relevant to the field of AI. You don’t have any reasonable algorithmic advantage for a reasonable amount of time, so realize that. All new AI modeling architectures are available to the whole world through published papers.

Tools and IDEs in an AI ecosystem for developing AI don’t set you apart either. All the tools and IDEs have free or budget versions. In addition to all this, you need a compute environment that is flexible, supporting both GPUs and CPUs. You need a decent AI development ecosystem, a decent compute environment, and decent AI modeling architectures, none of which are hard to acquire. As a result, you can compete with other companies, including very large technology companies.

Public Data Sources

Regardless of the industry you’re in, AI models are not developed solely on public datasets. You may boost your AI models with publicly available data sources, but your organization doesn’t have an edge if AI models are built only on public data sources. Your competitors can develop these models as well.

On the other hand, publicly available datasets are extremely important for advancing AI: MNIST, ImageNet, and many others have played a pivotal role in deep learning. When possible, consider opening your data sources to AI researchers outside your organization to advance the field of AI.

A Single Source of Truth

If you are not yet an AI-fueled business, you likely have multiple, disconnected data repositories rather than a single source of truth. You may think, “This is not too bad,” but there are many problems with multiple data repositories. Let’s start with the obvious one: it is not possible to combine multiple sources to gather more useful information. Another obvious problem is access rights; there will be many engineers and researchers at your organization who have access to some data resources but can’t access others. Imagine you acquire a third-party dataset that enables an AI project. To use the new data, you must join it with your existing data sources. Joining with each repository is a painful experience. For that reason, in practice, you will likely combine new data with only a couple of data repositories, which limits the AI training process.

Generated Data

AI-fueled businesses develop AI models using data generated by applications or stored in logs. Customer data, customer surveys, and third-party data sources are also used in AI model training. All this data tends to be continuous in nature, meaning there is fresh data available at some regular intervals. While building a product, you must pay close attention to the data it generates. Generated data is specific to your organization; it is a differentiator that helps you create better insights and better models.

You might come across data files that are not part of the production repository, which can make their way into AI modeling. It is a mistake to perform AI modeling on data that is not part of the production repository. It breaks some important rules, as repeatable experiments are possible only if you can locate training data in a production database. AI experiments that can’t be repeated are not meaningful. As a rule, store all your training data in production repositories.

Culture

What is an organization’s culture? It is the leadership style, work environment, policies, practices, and—above all—the day-to-day behaviors. You may wonder, what are day-to-day behaviors? They are a collection of how responsibilities are assigned, how feedback is given, how meetings are conducted, how conflicts are resolved, how innovative ideas are shared, how promotions are made, and many other things that come across while conducting daily business. An organization’s culture shapes and forms its AI culture.

If you believe you are in a “toxic” culture with a lack of inclusion, growth, opportunities, respect, transparency, and responsibility, you can’t be successful in your journey toward becoming an AI-fueled business.

If your company’s culture isn’t toxic, but lacks some elements such as support and enthusiasm, this will be detrimental to your AI journey. If you don’t have a good organizational culture, you first need to address that, which might not be easy.

In this section, I assume you have a good organizational culture to start with. I will elaborate on additional traits beyond a strong organizational culture for successfully becoming an AI-fueled business.

Setting the Right Expectations

Before covering what you should do, I will tell you what you shouldn’t do. You shouldn’t have large revenue expectations from AI at the beginning of your journey. You should understand that AI investments pay better dividends in the mid- and long-term.

Of course, you must define dividends for your AI-fueled business. Is revenue all you care about? Or are the following important for your organization: expanding your market, training your existing workforce in AI, and hiring AI executives, such as a CAIO? What does success look like for your AI-fueled business? Only you and your organization can define success.

You also must define mid-term. Is it the next 12 months? How about long-term? Are you referring to 24 to 36 months as long-term? For an AI-fueled business, return on investment starts at the mid-term and peaks at the long-term. If your organization keeps investing in AI, returns increase exponentially, and you may even move from one peak to another.

The expectations you have for AI in your business determine whether your organization succeeds or fails. For that reason, it is important to set the right expectations—to be more precise, you must set realistic expectations. Of all expectations, financial expectations consistently top the list; however, other expectations link to finances as well: how many new products and features can you develop with AI?

You will face obstacles in your AI journey. For each obstacle you overcome, you are building expertise that will become handy later. How long does it take to develop an AI product? How long does it take to hire an AI manager or an AI individual contributor? Along the way, you will build better intuition for solving AI-related obstacles. In a couple of years, you can also compare your organization with AI to your organization without AI. You’ll find more on this topic in the section “Pitfalls” in just a bit. Considering the pros and cons of AI, most organizations choose to stick with it due to its numerous benefits.

Learning from Failures

Resilience and perseverance are woven into the fabric of our society. Learning from experience builds resilience, which is a very important tool for becoming successful in your AI journey. Unlike conventional corporate communications, feel free to talk about what didn’t work for you. Reach out to stakeholders internally first and listen to their suggestions. See if they have experienced similar problems and find out how they overcame those problems. Becoming a learning machine is the most important piece of an AI-fueled business culture.

It may have been a common practice at some organizations, for some teams, for some managers, to sweep problems under the rug. However, note that this is a significant missed opportunity. It is a much better choice to place problems under a microscope without blaming anyone, to truly understand what went wrong and how issues can be corrected in future experiments. This is what successful companies do: they learn from failures; they are learning machines.

AI Experience

There are good books on AI, machine learning, data science, analytics, and related fields. Most of these books are written by academicians or consultants, and they do a good job from an academic or survey perspective. But these authors don’t have firsthand experience; they haven’t done it themselves, and they lack practical, useful information. You need boots on the ground to understand what AI development truly is. You need to build firsthand experience.

AI experience is an evolving concept based on individual experience as well as team and organizational experience. Once you start accumulating experience, you will hear bits and pieces about other people’s experiences. Some will relate their good experiences, while some will only mention their bad experiences. Both bad and good experiences are useful, however. Remember that you need to learn from mistakes. By reading this book, you are one step ahead of the competition, but you must start building your own experience. The experience you build becomes part of your organizational culture.

Pitfalls

The following pitfalls will slow you down or even stop your organization from being an AI-fueled business. They range from hiring the right personnel and setting expectations to establishing AI governance.

Avoid Know-It-Alls

Know-it-all types can be individuals or managers. However, it is worse if you have a know-it-all executive manager making unilateral decisions for your AI-fueled business. A know-it-all doesn’t have any problems making engineering, product, sales, or marketing decisions unilaterally. Worse, they don’t take any responsibility when something goes south, but they take all the credit for wins and successes. Avoid know-it-all types at your organization at all costs. It is simple to spot them, but it is not easy to get rid of them.

Avoid Muddying-the-Waters

Get your timeline in order. At your organization, mark the era before AI as BAI (Before AI, similar to BCE) and AAI (After AI, similar to CE) from year one with AI onward. It is important to get the stories right regarding what you had at your organization in the BAI era. Once you have AI models in production, two things may happen that can muddy the waters, potentially clouding the success of AI.

The first item concerns products. Some will say you already had the same product in BAI, so AI modeling didn’t add any value. They will conveniently forget to tell you that whatever you may have had was not working well, you received multiple customer complaints, and as a result, you lost customers. With the new AI implementation, you have a new product with the same name (unfortunately), but it works much better and attracts more customers. This criticism may come from any department within your organization, but it is likely to originate from the product management team. It is a good idea to have some naming conventions established around it—for example, if you had meter data management (MDM) in BAI, it is a good idea to tweak the name into Next Gen MDM in AAI for differentiation.

The second item concerns finances, especially revenue: you must price AI appropriately. Once AI is bundled with non-AI products, it is impossible to know how much AI is sold at your organization. Sales might even think and claim that AI doesn’t bring extra revenue due to cumbersome bundling dynamics and details. If you want to show that your organization’s AI investment pays off, it is best to either sell AI products solo or, if you have to bundle, don’t mix and match AI with non-AI.

Another note on pricing: AI products are the most difficult to build and get right. Don’t even think about giving them to your customers for free in a bundle, as is typically done to increase sales. It is well known that customers can use free AI products, but they won’t fully appreciate them. Free and open-source software has been improving the field of AI remarkably. I support contributing to both, but giving something away for free only to your customers is not the same as free software. If your organization decides to offer free open-source software (to everyone), that can be a huge differentiator that positions your organization very favorably.

Avoid Overpromising

GenAI, in particular, and AI in general have been game-changers. The constant emphasis on how to use AI to improve oneself, one’s organization, and daily life is brought to our attention through mainstream newspapers, magazines, podcasts, and social media. There is a lot of truth in these, but there is also some hype around each AI topic.

Riding the current wave of AI built on large foundation models will definitely inflate your expectations. Having similarly inflated expectations of your organization might sound like a natural extension, but it is also wrong. Although the underlying theory and technology improve at blazing-fast speeds, there are still problems to overcome. The most important thing is to discover the most suitable use cases for your organization. First, identify realistic use cases, and then forecast an increase in your customer base and revenue expectations.

Avoid Lack of Structure, Lack of Process, and Lack of Responsibility

Reading this book shows that you are taking organizational structure for implementing AI seriously. Once you have a functional structure in place, you need to follow best practices to utilize the organizational structure most efficiently. Both structure and processes will evolve over time, which is natural. Most AI work spans over multiple teams: in these projects, who is Responsible, who is Accountable, who is Consulted, and who is Informed (RACI) is not immediately apparent and doesn’t follow the reporting structure. RACI must be clearly defined for the successful execution of especially cross-functional projects.

Keys to Success

In this section, I lay out the most important traits that will put your organization in the fast lane for becoming an AI-fueled business.

Follow an AI-First Approach

As an AI-fueled business, you have the responsibility to consider AI solutions first. After thoroughly understanding each problem, consider pushing forward with AI solutions by carefully evaluating the pros and cons. Once you have an AI setup in place, it is both easier and cheaper to implement and maintain additional AI solutions. In some rare situations, a non-AI solution may work equally well, but by providing a non-AI solution, you’ll need to realize additional costs of implementation, deployment, and maintenance, which will be premium expenses. In other words, if you are an AI-fueled business with a large portion of solutions in the AI domain, you will pay a premium to implement and maintain non-AI solutions.

To put this interesting and somewhat unexpected concept into perspective, remember the disaggregation AI models I shared previously. All the disaggregation models (about 15 in total) are based on deep-learning architectures at Oracle Energy and Water. For all except one, a powerful deep-learning architecture is a necessity. The model, which didn’t really need a powerful deep-learning architecture, is “always on,” meaning devices are continuously powered on or on standby. Examples of always-on devices include Wi-Fi routers and modems, smart home hubs, TVs, security cameras, and smart thermostats. At Oracle, we implemented a statistical model for always-on that was very different from all the other models in the disaggregation family. Implementation of the model was straightforward, but keeping it up to date and maintaining the model turned into a nightmare. It needed special attention since it didn’t belong in a deep-learning architecture. Although it was not required, we reimplemented the always-on model with deep learning. Doing so made many tasks shareable and therefore easier, such as data curation, retraining, reevaluation, and deployment.

Hire AI Experts

A big part of education is general-purpose education, which creates awareness of AI. General-purpose AI education is necessary, but not sufficient. You must hire AI experts for strategic positions at your organization. Real-life AI development, deployment, and monitoring require real expertise based on education and experience. You need people who have seen it and done it before, preferably many times. Hire AI experts who can offer many products, both for business and for customers.

In our current environment, some people call themselves experts after listening to podcasts and reading newspapers and magazine articles. Remember that you must go beyond the hype. It is never a good idea to let so-called experts lead teams, departments, or organizations. But this is more common than you might think. Self-proclaimed experts create chaos by starting and stopping projects without any consultation. They kill the motivation of good employees and drive them out of your organization.

Newly hired AI experts have ample opportunity to share what is working well for your organization, and they also point out areas for improvement. As an aspiring AI-fueled organization, be open to feedback. In the absence of AI experts, if you have in-house grown talent that doesn’t come from the AI domain, you may end up doing more of the same. To further emphasize this point, imagine a manager who has been working on analytics for the last decade. You will have a hard time switching this person to AI. This manager will likely support a significant amount of analytics under the guise of AI.

Implement an Open and Inclusive Decision-Making Process

Most decisions improve when open discussions involve a group of people representing multiple teams within your organization. Make sure that everyone has a chance to contribute and is heard. Meeting leaders should ensure that everyone has a chance to speak. Indra Nooyi, in her Masterclass, puts it beautifully: there should be a person who runs a meeting, and this person doesn’t need to be the most senior person in the room. This person should allocate time for people who didn’t speak much by nudging them: “I wonder what you think about this topic?”

Some people may want to take over these discussions and force their opinions on others. If you have a frequent offender, think about limiting their attendance or participation.

Some decisions will be made single-handedly, but when making them, try not to upset a large portion of the organization. For example, don’t bring on the VP of Marketing as VP of Engineering, don’t bring on the Director of Product as a Senior Director of Engineering, and don’t form teams without talking to other teams at your organization.

An Anecdote

Vanessa Richter has been a sales executive for more than a decade at Oracle’s Opower Platform, which is part of Oracle Energy. Vanessa doesn’t see herself as a salesperson. She told me, “I am helping our customers; I match our customers’ needs with our products.” I have been working with Vanessa for several years. She is always thoughtful, well prepared, and available. She asks excellent questions that reveal important information. Working with Vanessa, it is clear that she loves her job and her customers.

I asked Vanessa to share the major parts of a successful sale. She told me her team works on identifying potential customers and determining their needs using publicly available data sources. It makes perfect sense for electric utility companies to share a lot in the public domain through press releases and other venues. Vanessa mentioned that the electric utility domain is quite visible.

Once customer needs are matched with products, contracting comes into play, which is another significant piece in the sales domain. A legal contract has many sections, but it has two main parts: Terms and conditions is the part that lays out the rules, obligations, rights, and responsibilities that govern an agreement. Terms of purchase focus on the purchased goods and services, including what a customer purchases, the duration of the transaction, pricing and payment, delivery of the goods and services, and all related terms. A contract is a long document. Vanessa mentioned that she gets a lot of help from internal resources, especially from the legal department.

During the long sales process, Vanessa and her team collaborate with regulatory, product, pre-sales, legal, and R&D organizations. Regulatory organization closely follows the rules and regulations in the electric utility domain. In some cases, these rules and regulations turn into needs and requirements. Product organization provides demos to sales as the initial engagement step. After seeing a good demo, customers are more interested and have follow-up questions. In this stage, pre-sales comes into the picture, and they help to answer more detailed questions about a product. If there are questions that are beyond the scope of pre-sales, they reach out to the R&D organization, including AI Research. The sales mechanism is a top-down approach that is meant to answer all the needs of a customer. Reaching out to the R&D organization is at the bottom of the process and is performed when needed. It is important to have this mechanism in place to be considerate of the time of the R&D and AI Research personnel so they have more time to innovate and implement.

I asked about how having AI products affects a deal. Vanessa told me that having AI in a product makes it a lot easier, and AI also fits into our value proposition well. She also mentioned that she needs more pre-sales support for selling AI products since there may be more detailed and technical questions that are beyond her team’s ability to answer.

We also talked about how AI can help sales teams in general. Vanessa reads many documents to learn about her customers and the industry. Besides driving sales, she also provides support to a team of salespeople. As a result, her time is limited. Vanessa is excited about using GenAI, especially LLMs, for researching, understanding, and analyzing documents.

Not all the sales initiatives go through. In some cases, a sale may fall short because the product isn't the right one for a target customer. A customer may point out another product in the market and may mention that it is superior to what is offered, regardless of whether the shortcomings are real or perceived. The Sales department brings money into an organization; they have a lot of leverage for obvious reasons. When they relate their conversations with customers regarding missing deals due to a lack of features or products, their feedback may be perceived as a demand. Vanessa thinks that Sales is responsible for bringing customer feedback, both positive and negative. And the corresponding salesperson who didn’t close a deal may have a strong point of view regarding missing features or products. This feedback must be heard, but it doesn’t mean that an organization rushes into new implementations. Strategy, in collaboration with the Product and R&D departments, decides whether to include these features or products in the plan.

At the end of our meeting, I asked Vanessa about her advice for new salespeople and what makes a good salesperson. Vanessa mentioned again that she doesn’t see herself as a salesperson. She shared a short story with me, highlighting the importance of helping a customer, even when there are no sales or financial gains. She said it is important to have curiosity; asking good questions opens minds and lets us discover the previously unknown, empowering us. Vanessa is hungry for information, which has served her well in her long and successful career.

Summary

In this chapter, I explained what an AI-fueled business is: AI-fueled businesses have AI at their core, which powers their business strategy, innovation, operational efficiency, products, and services. I also provided examples from well-known AI-fueled businesses: Google, Netflix, Amazon, Tesla, Uber, and Airbnb. Some of these companies are technology companies with technology at their core, some are not. Regardless, each of them uses AI extensively.

I explained the building blocks of an AI-fueled business: people, product, process, technology, and culture. People in your organization form teams, collaborate, celebrate successes, undergo change, and sometimes fail. Among all the building blocks, people are the most challenging to get right, but also the most important part. Investing in people makes your organization better. I introduced the very important concept of “embracing failure” and shared the importance of change management.

AI-fueled businesses are recognized by their distinctive products, and AI is the most powerful tool in their arsenal. AI-fueled businesses develop AI-first products that are loved by their customers, which may be individuals or other businesses. I shared the importance of data; without data, it is impossible to build AI products. I introduced ethics committees that guide data and AI model development processes. Finally, I emphasized the importance of continuously monitoring AI products. AI product governance is based on an organization’s resources and risk tolerance.

AI-fueled businesses follow efficiently designed processes for product development. Processes are designed to control risk and streamline work. In an AI-fueled organization, each AI team has expertise in its corresponding domain (for example, data, research, deployment, or production). Still, processes can be cross-functional by spanning the entire organization and linking regulations and recommendations.

I also covered important tools for development within the technology component. Open-source tools lead the way by enabling both large and small companies in their AI journeys. Similarly, academic advancements are available to all; the best AI modeling architectures are available around the globe just a few hours after they are published in academic archives.

Consolidating all data sources into a single source of truth helps both large and small companies avoid unnecessary searching. A single source of truth is the first step in trustworthy AI.

Finally, I introduced culture, which can be summarized as the leadership style, work environment, policies, practices, and day-to-day behaviors within an organization. Setting the right expectations, learning from failures, and building AI experience—both at the team and individual levels—are important attributes of an AI-fueled business.

Chapter 3. How to Structure the AI Organization

To benefit from the AI wave, you must have a matching AI organizational structure. You may be part of an organization that has some AI talent, but the structure may be somewhat haphazard; AI talent at your organization may be scattered around without any clear structure.

In this chapter, I explain how to build an AI organization in small and medium businesses, scale this structure to very large organizations, and run AI organizations effectively.

Very little is known about the AI Center of Excellence; there are no good resources on this topic, and most of the information comes from experience in technology companies. I have built an AI Center of Excellence at Oracle. I cover the AI Center of Excellence in detail, outlining its overall mandate and explaining its key pillars, including Best Research Practices, Human-Centered AI, Data, and MLOps.

Hopefully, after reading this chapter, you will know a lot more about how to construct an AI organization and how to build an AI Center of Excellence.

Hierarchical Structures

Hierarchical organizational structures tend to work well for both medium and large companies. In a hierarchical organizational structure, there are multiple levels of authority with a clear chain of command. If there are multiple lines of business, a similar level of authority is defined within each line, and the head of each business reports to the CEO.

In small companies and startups, the organizational structure is usually flat, which works well because everyone knows what everyone else is working on. In a conventional hierarchical organization, each AI organization takes form under a line of business (LOB), below the corporate level. I call this semi-holistic from an AI perspective, since AI components are not shared across lines of business. For example, two AI Research teams may be developing similar AI models, and they wouldn’t be aware of each other’s accomplishments or lessons learned since each team works in isolation.

The holistic AI approach aims to solve the problem of isolation with a dedicated C-level executive who links all AI efforts under each line of business.

Holistic AI Approach

An emerging C-suite title is Chief Artificial Intelligence Officer (CAIO). A CAIO oversees AI strategy development, research, innovation, and deployment activities. A CAIO decides what emerging AI technologies to follow or implement in products and services. A CAIO also decides AI areas for investment, what to innovate on, and what to contribute to. The CAIO role is relatively new and has higher adoption rates in Europe than in the United States, but, in recent years, companies with CAIO titles have increased threefold.

Among large technology companies, SAP adopted the role of CAIO by appointing Dr. Philipp Herzig in February 2024. Dr. Herzig describes his role in social media appearances, which involves developing and implementing an AI strategy, ensuring the coherence of AI developed by multiple teams within the company, keeping pace with rapidly evolving AI, and coordinating AI initiatives with business units. During these interviews, Dr. Herzig mentioned that he reads newly published papers on AI. Dr. Herzig holds Bachelor’s and PhD degrees in computer science from Dresden University of Technology, so he is well situated to read and understand highly technical AI papers. Dr. Herzig provides a good example of a C-level executive who is both highly technical and practical, understanding the value of AI in products and services. Dr. Herzig has been serving as both CAIO and CTO at SAP since January 2025.

Around the world, governments promote the development of ethical, safe, and trustworthy AI systems, such as the following examples:

• US: National AI Initiative Act

• EU: AI Act

• United Kingdom: National AI Strategy

• India: National Strategy for AI

• Japan: AI Strategy

• Canada: Pan-Canadian AI Strategy

• Singapore: AI Strategy

• South Korea: National AI Strategy

I am familiar with the US National AI Initiative Act, which has been led and promoted by the US Presidential Office. AI Initiative documents are well-written and align with state-of-the-art developments in the field of AI. No single person writes these documents; instead, each document is the product of a large consortium of experts from the public, private, and university sectors. It is also important to note that these documents contain a wealth of practical information and wisdom.

An executive order issued on December 3, 2020, required the appointment of Chief Artificial Intelligence Officers (CAIOs) at US federal agencies. This order is the same as the Executive Order on Promoting the Use of Trustworthy Artificial Intelligence in the Federal Government. To provide a couple of examples: the Departments of Energy, Education, Commerce, and Homeland Security, among others, each have a Chief AI Officer.

A Chief AI Officer is still relatively uncommon in US companies, but it is growing rapidly. Some large technology companies, such as Google and Microsoft, have dedicated C-suite positions for AI. Some retail and financial companies have also adopted the AI role. Only 5-10% of Fortune 500 companies have a dedicated CAIO role. Around 50% of Fortune 500 companies assign AI roles to a Chief Data Officer, and 30-40% assign them to CTO or CIO roles.

CAIO is a fairly new occupation that some large companies have adopted. In some cases, an older and somewhat more established role of Chief Data Officer or Digital Officer (CDO) assumes responsibilities. As data is central to their organizations, a CDO also assumes the role of AI and Analytics by aligning their company around AI. A good example is ADP’s ex-CDO Jack Berkowitz. In the September 27, 2002, episode of the podcast “Me, Myself and AI,” Jack discusses how part of his job was overseeing AI at ADP. In most cases, a CAIO or a CDO has a small team and receives help.

A CAIO or a CDO oversees AI operations by making sure that:

• An updated AI strategy matches the mission, product vision, product strategy, and business objectives.

• AI research, deployment, and production are coherent, modular, extensible, and compatible.

• AI offerings within the company are well documented and communicated.

• Emerging AI technologies are evaluated and adapted.

Having all these at the very top level is great, but insufficient. Within each line of business and in each department, you need a Head of AI who is responsible for that line of business. The head of AI not only communicates within a line of business but also with other heads of AI within a company. Ideally, AI-related activities, such as AI research and deployment, are organized under the Head of AI. This person is also responsible for AI communication and AI product vision. Unlike a CAI or a CDO, the Head of AI’s responsibility is scoped within a line of business. For example, a Head of AI might be responsible for the Energy line of business.

The Head of AI’s hierarchical level may vary according to your organization: SVP, VP, or Director—it doesn’t matter much, but it is important to note that the Head of AI has a seat at the business’s executive leadership table. If that is not the case, it may mean that the business doesn’t value AI. The Head of AI should maintain continuous communication with other heads, including those from sales, delivery, engineering, customer service, and others.

Figure 3-1 illustrates a simple yet effective organizational structure. In this structure, problems and conflicts may arise, but they are resolved in a coherent manner. In this organizational structure, AI research, deployment, and production are implemented efficiently and uniformly, and lessons learned within one team are applied to the other teams.

Figure 3-1 AI organizational structure

The organizational structure shown in Figure 3-1 is currently emerging but is not yet common. Lacking this organizational structure inhibits a company in various ways: AI model development and deployment efforts are duplicated by multiple teams, mistakes are repeated, and dependencies are unmet. In short, there is abundant waste in the form of reinventing the wheel, reencountering and resolving the same problems in different, incoherent ways. With the CAIO's guidance and support, the Head of AI implements structure, rules, and guidance.

Although virtually all companies acknowledge the benefits of AI, most don’t acknowledge AI as a department with multiple teams. Hence, AI is scattered under engineering without a Head of AI. A re-organization might not consider AI as an atomic unit, or there may be some political reasons within an organization to distribute AI—for example, executive managers might have some control over this very powerful technology. Or some companies might think AI isn’t yet mature and hasn’t proven its usefulness; therefore, it isn't treated as a first-class citizen. Figure 3-2 depicts this very common situation by showing scattered AI teams within a line of business (LOB).

Figure 3-2 A product ecosystem with AI

Figure 3-2 has a couple of disadvantages worth unpacking. First, and most obviously, without a Head of AI, there is no coherent AI mission and vision. Without a clearly defined and executed AI vision and mission, a line of business works on projects haphazardly: a line of business doesn’t have clear priorities for even very large AI branches, such as GenAI. Each team or line of business performs its own development and testing, depending on its appetite for new technology and availability.

Misalignment is common in organizational structures, as shown in Figure 3-2. Each VP has varying objectives and key results (OKRs) and responsibilities. As a result, an AI development activity for one team might not have a matching OKR in another team. This happens quite often in this type of organizational structure, and in this case, other teams ignore major dependencies to satisfy their own OKRs. You can also call this piece “politics” if you prefer, but misaligned reporting structures bring misaligned OKRs and responsibilities. Even with the best intentions, most experienced managers and engineers find it very challenging to overcome these misalignment problems.

Due to differing reporting structures, it is very likely that planning calendars won’t align either. Some teams won’t do any planning, and some AI teams will align their planning with their reporting structure but not with other AI teams. For example, an AI research team may plan several months later than the AI Deployment and AI Production teams. Dependency tickets won’t be satisfied until the next quarter due to planning misalignment. AI research teams often face unmet dependencies for data curation and other engineering tasks directly related to AI model development.

A holistic AI approach, with a CAIO guiding all AI activities, provides the best reporting structure. In the absence of a CAIO, the second-best option is to have a Head of AI for each line of business who guides all AI activities by closely coordinating with other department heads. If, for some reason, neither is possible, an AI Center of Excellence can help bring order and structure.

AI Center of Excellence

In the ideal case, a CAIO leads AI efforts in an organization by coordinating with each line of business’s Head of AI. Similarly, a Head of AI leads AI efforts within a line of business. Each AI team within a line of business is aware of what other teams in that line are working on. A Head of AI is knowledgeable about projects across different lines of business and communicates this information to all AI teams within the respective line of business. This structure creates strong synergy, reducing waste and duplication.

In the absence of a CAIO role, other roles such as CDO, CTO, or CIO can assume top-level responsibilities. In the absence of a Head of AI role within a line of business, an AI Center of Excellence can coordinate AI. A large organization can situate an AI Center of Excellence in strategic locations: most commonly within a line of business or at the corporate level. Large organizations can have multiple AI Centers of Excellence, one for each line of business. Some lines of business may not have an AI Center of Excellence for various reasons.

In this section, I will focus on the most common use case, an AI Center of Excellence within a line of business. In some cases, there may be both a Head of AI and an AI Center of Excellence within a line of business. Unlike a Head of AI who has a clearly defined reporting structure, an AI Center of Excellence commonly plays an advisory role.

I’ll outline the need for an AI Center of Excellence from an AI technical perspective, starting with how AI teams work and the key pillars that make up an AI Center of Excellence.

An AI Research team conducts many experiments for developing AI models. For each experiment, multiple metrics are logged and compared. Each experiment can differ from the others in the following respects: model hyperparameters, model architecture, algorithm used, training dataset, and so on. More experiments generate more AI models, which in turn means more models with high accuracy metrics. AI researchers don’t blindly run many experiments; instead, they check the results of previous experiments and make necessary changes to a new experiment, expecting better AI models. The quantity of experiments can measure the AI maturity of an organization run by AI researchers. Running multiple experiments turns into accumulated knowledge, which can be seen as valuable insights for the AI model and industry data. A set of experiments guides AI researchers and AI research teams, and prepares them for future AI model development tasks. Not sharing this useful information and these insights with others creates an island of experimentation. In this case, an island can be an AI researcher, but more often it is an AI research team.

An island of experimentation has a negative connotation, referring to a team that performs AI tasks independently without sharing information with others. Islands of experimentation are more common than you might think. There may be multiple teams working on similar initiatives, even in similar time frames, without being aware of each other’s work or progress. Repetitious work might include AI modeling, AI deployment, AI production scoring, or some other engineering task. A variation of this problem occurs when an AI research team starts a new project, but another team has already conducted a similar project. The previous work didn’t produce good results; hence, it wasn't documented or publicized. The new project team is unaware of the methodologies, datasets, and assumptions used in the old work. If they had been aware of the previous work, they could have seen which models and data were tested. This information would help them determine whether to proceed with the project. Any form of repetitive work is inefficient in an organization and should be avoided.

As Figure 3-3 illustrates, an AI Center of Excellence has many tasks and duties; its primary role is to serve as a bridge between business units and AI teams. In doing so, an AI Center of Excellence avoids duplication of effort and guides a project to the most knowledgeable AI team and researchers. In other words, an AI Center of Excellence strives for efficiency.

Figure 3-3 Pillars of an AI Center of Excellence

The AI Center of Excellence sets standards and best practices that work well for your line of business. The starting point for these is a globally accepted set of AI rules and best practices, but a line of business doesn’t have to adopt all these rules at once. One option is to set up a practical goal and make progress toward that goal incrementally.

The AI Center of Excellence is the mental muscle of your organization. Acting as a center to accept and distribute the work, it also logs what has been accomplished before and how it was achieved by listing the people, skills, and tools used so that new project members can learn from the previous experience, as well as new technological advancements. There are many important pillars in an AI Center of Excellence; each pillar focuses on internal teams by keeping a close eye on the state of the art. State-of-the-art AI changes rapidly, is abundant, and takes place outside your organization. Therefore, it is important to keep an eye on external developments outside your company. This takes time and expertise; the main task is to determine how to tailor the latest technology to fit your organization. The most common pillars of an AI Center of Excellence are Best Research Practices, Human Centered AI, Data, and MLOps.

An AI Center of Excellence Mandate

Why might you need an AI Center of Excellence? You are part of a line of business with various data sources and AI teams. You have some experience with AI, including model training and validation, model deployment, and model scoring in production. You align AI with your company’s objectives by solving complex customer problems and delivering better, faster, and more cost-effective solutions. You've taken baby steps with AI; now you need to move faster by scaling AI within your line of business, but you're unsure how to do it. You notice some inefficiencies, some of which are specific to AI. When you talk to AI teams about business problems, they say they have solutions, and they produce some presentations to back up their claims. But these teams don’t know about the details of AI modeling work, including training data, inputs, where AI models reside, and when a proof of concept (POC) was developed. You constantly find yourself in demo mode, and you want to get out of it. At a high level, for the following cases, you need an AI Center of Excellence.

First and foremost, the duty of an AI Center of Excellence is to serve as a bridge between business and AI teams. There are many reasons this connection is needed: business units and AI teams don’t quite share a common language, business units often don’t describe problems, both teams prematurely jump to a solution, and AI teams don’t know about the work that's already been done in a line of business.

If you witness some AI skeptics at your company turn into believers, you want to gain their support. You must steer your company’s destiny and turn it into an AI-fueled company. You may think that you can solve every problem with AI, and you may be right, but you must put a system in place. An AI Center of Excellence establishes a system that supports AI work within a structured framework.

At a high level, an AI Center of Excellence also leads AI change management at an organization. Often, AI projects fail due to a lack of support from conventional thinkers within the organization. An AI Center of Excellence can provide quarterly updates on accomplishments and overall progress. By doing so, it increases support within the company, thereby increasing the success rate.

You have developed AI models that are in production as products, features, or insights. You can attest to the importance of AI models. You had a taste of AI victory and want more. Money and resources are tight at your organization, and you want to get the best bang for your buck. An AI Center of Excellence can organize the resources and investments.

You have data resources and want to use them to build AI models. You want to monitor these models in production and ensure they deliver value to your customers.

Best Research Practices

There is positive public sentiment toward AI, in part due to overwhelming interest in generative artificial intelligence (GenAI). Many benefits provided by large language models (LLMs) directly affect our daily and work lives. These models are attracting significant public attention, as they are featured in popular newspapers, TV channels, and on social media. GenAI models offer both free and subscription services, and it is estimated that several billion users are benefiting from GenAI models directly or through integrated search engines. Large technology companies invest a lot of resources in building and testing foundational LLMs such as ChatGPT, Llama, Cohere, Gemini, and others.

Recently, a Nobel Prize was awarded to well-known AI researchers for their contributions. In 2024, Geoffrey Hinton shared the Nobel Prize in Physics with John J. Hopfield for foundational discoveries and inventions that enable machine learning with artificial neural networks.¹, ² There is no Nobel Prize in computer science or AI categories; hence, the award was presented in Physics. AI researchers know well that research on neural networks, which forms the foundation of GenAI models and many other deep learning techniques, hasn't always gone smoothly. Neural networks have had a couple of long winters and setbacks.

¹ https://www.nobelprize.org/prizes/physics/2024/hinton/facts/

² https://www.nobelprize.org/prizes/physics/2024/hopfield/facts/

You may wonder why we need the Best Research Practices pillar? In short, this pillar achieves three primary outcomes: establishing and implementing best research practices, following external research work, and conducting your own research. All these outcomes are tailored for your organization.

About 1,500 new publications are estimated to appear daily in the AI domain. This large number includes peer-reviewed work, preprints, and industry research. It is overwhelming for an AI researcher to stay up to date with AI advancements. The Best Research Practices pillar helps identify relevant papers from relevant conferences, journals, and industry magazines. The Best Research Practices pillar also includes brainstorming sessions to discuss research papers.

This activity aims to identify the state-of-the-art that best suits your business’s needs. You must judge whether a methodology is sound and whether the experimental datasets are representative of the problem domain. Once you spot credible evidence, you can chase the related code, tools, and libraries, and do your own testing on the same dataset to see if you can reproduce the same results. If that is the case, and you want to take it to the next level, you may want to expand the datasets to see if you get similar good results. If everything checks out, you may consider improving the existing implementation to achieve better results on your own datasets. Improvements vary depending on your line of business and your organization's appetite for conducting internal research. A very small number of companies are set up to conduct fundamental research.

Remember that, most often, you are not dealing with pure technology problems; you need to pull in product teams to see whether they are interested in offering new products or features built on what you have spotted. This is a highly effective process that brings together multiple teams and leads to quick wins.

The first two items focus on digesting research conducted outside your organization and applying it within your organization by improving it, storing this knowledge for further use and dissemination. Breadth and depth depend on your organization's needs and the skill sets it has.

We must build resilience and grit by embracing and learning from failures. The concept of failure is not fully embraced in most corporate environments. Some people think you may be wasting company resources on something that doesn’t work. They point out what works at the moment and suggest allocating more resources to it. Learning from failures is an important concept that leads to the third major item: establishing best research practices. Only after you know what works and what doesn’t work for your line of business can you put the best research practices together. It is also important to search this topic to discover whether large technology companies share their insights. It is hard to find these kinds of insights. Even though it is not a good idea to replicate their experience, it is a very good idea to learn from their experience and insights.

Human Centered AI

An important paradigm that is receiving a lot of positive attention for the right reasons is Human-centered AI (HCAI), which grounds AI on human users. HCAI reminds developers and designers why they develop AI models. Human-centered AI is a collection of practical concepts that puts humans at the center of the AI model development cycle.

Interactions between an AI model and its user should be two-way communication. Instead of an AI model telling a customer what to do, it should list options and help the user decide. It is important to keep in mind that the user is the one who makes the final decision. AI models help users by providing useful predictions, forecasts, and insights.

A user of an AI model may disagree about some insights, so the consumer should be able to provide feedback. The granularity of the feedback ranges from coarse to fine. A somewhat coarse form is “I don’t agree.” More detailed feedback can be collected in the form of “I don’t agree; the correct insight is ….” The feedback mechanism is very important, and puts the human in the center. Although receiving and storing feedback is technically not difficult, this is a significant shift in mindset. Another substantial effect of this mindset is that user feedback provides new data, which in turn should be used for retraining improved AI models.

Let’s make this concept more concrete by providing a product example from the energy domain. A home energy report provides a detailed report of a household’s energy usage. AI models estimate the energy usage of household appliances, such as HVAC systems, water heaters, EVs, and clothes washers and dryers. Utility companies provide home energy reports to their customers so customers can gain insights into their electricity usage. They are better informed to take control of their electricity usage. Figure 3-4 provides a sample electricity bill without detailed breakdowns, and Figure 3-5 provides a sample home energy report based on AI models (especially deep-learning models).

The most basic case of feedback is to provide insights from AI models to customers directly without any feedback loop. In Figure 3-5, a home energy report shows its customers what they used, without any feedback mechanism. A home energy report provides a breakdown of the energy cost of the top three appliances. Note that the electricity bill in Figure 3-4 doesn’t provide a breakdown of appliances, only total energy usage, but does offer a breakdown of services such as distribution service charge, electricity supply charge, sales tax, and taxes (state, local, and county).

Figure 3-4 A example electricity bill

Figure 3-5 A home energy report

In Figure 3-6, a customer may challenge insights provided by AI models for various reasons by not agreeing. In this case, customer feedback is high-granular but lacks details; a red X indicates the customer disagrees with the AI model's estimate. The feedback is logged and stored in a data store for further use.

Figure 3-6 A home energy report with some user feedback

More granular customer feedback that goes beyond “I don’t agree” is more useful. In this case, the customer corrects a value provided by a home energy report for a reasonable motive. Figure 3-7 depicts this situation, where a customer disagrees with the water heater estimation and provides a correction: “Gas Water Heater (not Electric); therefore, corresponding electric usage should be $0.” As another example, a customer may correct EV (electric vehicle) Charging (not shown in the figure) by checking the information in their EV app. Yet another example might be a customer measuring appliance consumption with a plug-in energy monitor and correcting the AI estimate by providing information from that device. This may sound like a rare situation, but it is becoming more common with plug-in energy sensors and with smart panels. Smart panels may provide a very accurate breakdown of energy consumption, serving as a reliable source of accurate usage measurements in a household.

There are many ways to gather user feedback. Unfortunately, most products don’t have a feedback mechanism. The first step is to enable feedback. Enable feedback and store it to learn from it and train better-performing, more accurate AI models. AI models are powerful, but we all know that they make mistakes. Collecting customer feedback is still very rare, but it is becoming more mainstream with GenAI-style feedback—for example, Like and Dislike buttons, followed by “Share any comments” feedback, or a free-form text box.

Figure 3-7 A home energy report with detailed user feedback

Data

Data is oil for the AI machine, and it requires special and constant attention. Data sources vary. Some may be internal to your organization, some may be externally resourced, and others may be internal to your line of business. It is challenging to find out what data is available within your line of business, especially at large companies. Externally resourced datasets are customer data, public data, and third-party data. For training purposes, most companies have rules and regulations that align with local, state, and federal regulations. AI models don’t need personally identifiable information (PII) or personal information (PI). Before training AI models, check rules and regulations to see what data you can use for training. Note that the laws and regulations are subject to change over time.

Data is a significant enabler; consider it in dynamic terms—data that is not available for training at the moment may become available later. This means that data searching is a never-ending quest. You may consider licensing data from third-party providers. The process of data acquisition includes the following steps:

1. Explore data sources

2. Hold meetings with third-party data providers

3. Explore the data dictionaries and sample data

4. Perform exploratory data analysis

5. Collect feedback from AI research, development, and product teams

6. Offer a business case for data purchase

7. Contract with a vendor

8. Obtain credentials for accessing data

9. Revisit exploratory data analysis to confirm that it matches the data dictionary and promises made by any third-party data providers

10. Ingest data and make it available to your line of business

Data acquisition may be a bit involved; you should have a dedicated point of contact who represents your organization. This person follows all the steps that I outlined previously. I led multiple third-party data purchasing tasks as the main point of contact. I always found the result of having a new data source very rewarding, even though it requires considerable time and effort.

Having data is important, but it is equally important to have a data platform, a single source of truth, to serve this data to your line of business: AI, application, development, and product teams.

The concepts of production environment and development environment apply to data as well. In most cases, data that resides in a production environment can’t be used in a development environment—a common-sense rule in large technology companies. Most useful AI models are developed in production environments using real datasets.

Useful synthetic data is becoming more common through advancements in technology. Some government organizations develop synthetic datasets. A good example is Commercial Building Stock (ComStock) data from the Department of Energy’s NREL division.³ As stated on the home page, ComStock is a US Department of Energy model of the commercial building stock, developed and maintained by NREL. It is hard, if not impossible, to reach out to real commercial energy data. ComStock provides a reasonably good dataset that is open to researchers and for commercial purposes.

³ https://comstock.nrel.gov/

Data Pillar oversees the data platform, utilizing existing data sources, acquiring new data sources, and following rules and regulations for training and deploying AI models.

MLOps

Machine learning operations (MLOps) streamlines and scales the development, deployment, monitoring, and management of AI models. MLOps is about automating the end-to-end AI lifecycle, which includes data curation, exploratory data analysis, model training and validation, model evaluation, model deployment, model scoring in production, model monitoring, and input data monitoring.

Automation is performed by jobs that make major tasks easier for AI researchers, so that manual development efforts are reduced; as a result, AI researchers can run more experiments. For example, data curation can be automated through a job that retrieves, organizes, and manages datasets. Automation turns raw datasets into feature sets and splits them into training, validation, and test. Data curation is an involved task that requires significant compute resources and is executed in a distributed fashion, such as with Spark.

AI researchers use Jupyter-style notebooks for exploratory data analysis. Although each dataset may need tailored analysis, there are plenty of commonalities between datasets in terms of functionality and software development processes. Jobs from an MLOps automation perspective should handle these commonalities.

Data curation is complex when working with real datasets that grow over time, which is the norm. On growing datasets, for retraining purposes, you can’t start data curation from scratch each time. The concept of ID-preserving datasets involves creating training, validation, and test datasets that continue the previously generated datasets. With ID-preserving, you can compare new AI models with those in production without worrying about data leaks from previous versions.

AI model development is experimental in nature. AI researchers must run many experiments to identify the most effective ones. For that reason, it is crucial to run jobs for training models and save results in a data store. Similarly, once you have many models trained, you need to have another job for evaluation—to compare metrics and uncover more details. In practice, the winning model is seldom clear; there will usually be mixed results, and a model will perform better on some metrics but not as well on others. Plus, running all models on new data is unfair, since only the new models are trained on fresh data. At the same time, running on historical validation datasets provides some insights, but it may not be an apples-to-apples comparison. Recently trained AI models capture recent customer behavior more accurately.

Both dataset generation and running training and validation are part of MLOps—these are major tasks in AI. Another major task in MLOps is monitoring AI models and input data in production. Scoring AI models in production without any monitoring is not a good idea. For that reason, the accuracy of AI models in production should be compared with real-time production. As AI models are developed, their accuracy metrics are stored. While AI models run in production, new accuracy metrics on production data are compared with what was obtained earlier in the development phase. The difference is known as drift. Large drift indicates problems with the AI models. There are several reasons for large drift, including changing customer behavior. Another type of drift may occur in the input data. Similarly, real-time production data statistics should be compared with the training data statistics—significant changes are a reason for concern. Each line of business should decide what percentage of drift is acceptable. Large drifts halt production scoring and require retraining AI models on new, recent datasets.

MLOps is an important topic that covers a lot of ground. There are good books written on the subject. When implemented correctly, MLOps saves time, scales AI model development, and provides a guardrail for AI model scoring in production.

Not having any MLOps at your organization is unacceptable; similarly, having each team in your line of business develop its own MLOps isn’t acceptable either. An AI Center of Excellence not only promotes the use of MLOps, but it also coordinates the development and uniform use of MLOps. An AI Center of Excellence ensures that there is no duplication of efforts, and all teams speak the same language.

Pitfalls

When assembling AI organizations, avoid common mistakes. The first and foremost is avoiding know-it-alls; these individuals, without appropriate qualifications, credentials, or experience, will muddy the waters, putting themselves on stage to garner credit without accomplishing anything useful. Information flows from many directions, and learning about AI feels like drinking from a fire hose. There is conflicting information as well, but taking no action won’t get you anywhere. On the opposite end of the spectrum, you should secure financial and managerial support at your organization before making AI announcements. AI is very technical, but AI is here to help us, your customers, and organizations. Think broadly, have AI governance in place, and, most importantly, think in business terms.

Avoid Lack of Technical Debt

AI is everywhere; it appears in mainstream media and social media. There is a lot of publicity about AI, with many articles and books written on the subject. GenAI and large language models especially have made their way into business magazines, including the well-known Harvard Business Review and the MIT Sloan Management Review. These two magazines combined have an estimated one million subscribers—a very large audience. Mainstream newspapers such as The New York Times, with around 10 million paid subscribers, also cover AI topics in depth.

It is straightforward to follow AI-related articles published at the Harvard Business Review, the MIT Sloan Management Review, and The New York Times. Managers reading these articles and similar ones may think they know it all, especially if they don’t have a technical background. Avoid having managers at the AI Center of Excellence with only a business perspective or common knowledge. The AI Center of Excellence has a technical focus; non-technical managers may be good administrators, but they don’t connect well with the AI Center of Excellence team. A lack of connection, combined with a lack of understanding, creates a chaotic, unstable environment where AI can’t flourish. At the AI Center of Excellence, the technical perspective comes first, which is tied to the business outcome. Without appreciating technical challenges and a good perspective on how to overcome them, an AI Center of Excellence loses a leg and becomes unstable.

Avoid Inaction

Some people can’t take action. Don’t be one of them. Similarly, some organizations can’t take any actions; don’t stay in one of those. Put your knowledge to work, find out how you can improve yourself and the organization in AI, and brainstorm with key stakeholders. Most importantly, take action because procrastination in AI is costly. No decision is perfect; you must make educated and reasonable decisions. More importantly, make it a habit to make small changes as you execute your decisions, steering your team in the right direction. Make it a habit to learn as you go.

Avoid Jumping the Gun

Don’t announce your ideas prematurely. As a manager, your role is to secure financial resources before making announcements. You may think you have a great idea, or you may genuinely believe that you will have the support of your peers and management. Early and premature announcements that are not followed through on are very demoralizing to the whole team; they chip away at your credibility.

As an example, suppose you love the idea of having an AI Center of Excellence, you spoke with your peers and management, and they are supporting you. It is too early to announce it at your all-hands meeting if you haven’t secured financial resources to set up an AI Center of Excellence. Worse, imagine that you made the announcement but didn’t follow through. Nobody will take you seriously from that point on. I love how Indra Nooyi explains this concept eloquently at her masterclass, “Leading with Purpose.”⁴

⁴ https://www.masterclass.com/classes/indra-nooyi-teaches-leading-with-purpose

Avoid Thinking Only in Technical Terms

AI is technical, and there is no way around it. But a common mistake many managers make is to think only in technical terms. If you want to be successful, you must also think in business terms: what are the business objectives, and how can you produce better, faster, and cheaper products? Think about your products and how integrating AI can attract more customers and increase customer satisfaction. You don’t need to have an MBA, but I suggest you read business articles. Companies that build a strong bridge between AI and business become more successful. And your AI Center of Excellence serves as that bridge.

Keys to Success

To reap the benefits of AI, you must have an organization that runs like a well-oiled machine. Ensure that you secure financial support before initiating AI projects. Support from high-level executives goes a long way. Being short-sighted in AI will not take you far; you must establish an AI vision that is both practical and groundbreaking. Remember, AI has many components; adopt an AI mindset to maximize its benefits and guard against imperfections.

Align for AI

To benefit from the AI wave, you must have a matching AI organizational structure. This means adapting your approach with a focus on AI. In general terms, change is hard, and AI change is no exception.

You may be part of an organization that has some AI talent, but the structure may be somewhat haphazard. AI talent within your organization may be scattered across the organization without a clear structure. You must rally for an AI hierarchical structure or to achieve cohesion with the AI Center of Excellence.

To reap the benefits of the AI boom, achieving cohesion within your line of business through AI must be prioritized first. It is hard to achieve meaningful AI gains without a focus on AI structure. Without a proper AI structure, your organization might achieve a few wins, but those wins will be both expensive and unsustainable.

Sponsor AI

As much as AI supports your products, features, and insights, you must support the AI development process as well. Both top-down and bottom-up organizational support are necessary and important for reaping the benefits of AI.

Top-level support starts at the C-level leadership at your organization and continues with the general manager. An executive sponsor is a high-level technical manager who likely reports to the top level at your organization. All AI projects need an executive sponsor who keeps up with technical developments, unblocks AI teams, and shares progress with top-level management.

Bottom-up support is arguably more important; for a successful AI project, you need fully invested engineers and developers. They have the skill sets and drive to take on AI projects with unwavering focus, addressing obstacles directly and pushing initiatives to success. It is also important to have backups of non-technical contributors from sales, marketing, and legal.

Establish AI Vision and an AI Strategy

Your line of business, similar to your organization, has both a mission and a vision. From your business vision, you have a product vision, and, aligned with that, an AI vision. AI vision is a high-level description; an AI strategy makes it concrete by providing details on how to get there.

A sample AI vision looks something like the following snippet:

“Empower our customers with responsible, scalable, and human-centric AI solutions that improve productivity and decision-making.”

And a matching AI Strategy snippet looks something like this:

“Investments in AI: Establish an AI Center of Excellence and collaborate with other lines of business at our organization. Drive innovation, present at well-known industry events, journals, and conferences.

Prototype new AI capabilities, allowing quick iteration and real-world testing of concepts.

Develop ethical guidelines and internal review processes to ensure all AI models meet transparency, fairness, and bias mitigation standards.”

Not having an AI vision and a matching AI strategy feels like going wherever the wind blows, and AI wind blows in every direction. AI vision gives you a clear direction and purpose, and AI strategy turns that vision into actionable steps.

Evolve into an AI Mindset

AI models come with their pros and cons. Unfortunately, many users expect perfection from AI. Be mindful that AI is advancing; it has flaws and is not perfect, as with all scientific fields. It is crucial to guard against imperfections in AI within products to ensure your customers are not adversely affected.

Openness to technology is essential, as mentioned earlier, and thousands of AI papers show rapid progress. You should make it a habit to be open to new developments, which requires both curiosity and continuous learning.

An Anecdote

Shannon is part of Oracle’s pre-sales department, which is also known as “solution consulting.” She is a double-major in environmental sciences and economics. Before joining Oracle, she was a consultant at Accenture’s utilities practice. She knows the utilities business inside and out.

I interviewed Shannon to understand her day-to-day job duties and how AI relates to her role. Her primary job duty is to provide support to sales for our products, especially AI products, where her expertise is most needed. She told me she needs fundamental knowledge of AI models, but she doesn’t stop there; she also needs to know about AI model training and validation. She has access to our products, and she is very knowledgeable about the results of AI products. By continuously “testing” AI products through use and queries, she provides much-needed internal feedback to our organization. She also receives external feedback during sales or after a sale to support a customer. Shannon is a big proponent of AI model retraining. She understands the concept well and brings up good examples to support cases for retraining.

Shannon supports sales by giving presentations and answering questions at meetings or afterward through email. Shannon has well-prepared presentations, and her audience commonly asks off-topic questions, which she handles skillfully. I’ve learned that off-topic questions are common; hence, it is important to have a detailed understanding of AI products.

I work with Shannon in our AI Insights Core Team. Shannon is very vocal and active in the core team by providing feedback and pointing out discrepancies she notices while using a product. Each core team has a focus area, and at meetings, they discuss problem areas and how to improve them.

Shannon also actively participates in crafting our white papers by providing customer perspectives, including what they would like to see and how these documents support the sales process. Her feedback on white papers is received with enthusiasm at the core team level, and it is invaluable.

It is not easy to be in front of customers representing AI products that have numerous details. I asked Shannon how she feels about it. She told me she loves it, tries her best to answer all customer questions, and, for the questions beyond her knowledge, she reaches out to the product and AI Research teams for help.

Summary

This chapter covered AI team topology. I introduced organizational hierarchies that fit naturally for AI management and execution, applicable to both medium and large organizations. In a conventional hierarchical organization, each AI organization takes form under a line of business, below the corporate level. A holistic AI organizational approach is introduced, with an executive at the top of the corporate level, typically a C-suite member, and business heads reporting to the C-suite. I provided examples from leading technology companies. I also shared the US Federal government’s approach to AI management. I talked about C-suite titles such as Chief AI Officer and CDO (which can mean Chief Digital Officer or Chief Data Officer). In the absence of these titles, I shared how a CTO or a CIO handles AI duties at the top corporate level. I also introduced Head of AI roles within each business unit, especially in large organizations, and explained how a Head of AI leads and coordinates all AI efforts.

For many reasons, reorganization around AI might be a bit messy and time-consuming. When holistic or semi-holistic AI management approaches are not possible, or not an option, the best option is to establish an AI Center of Excellence. An island of experimentation has a negative connotation, which refers to a team that performs AI tasks independently without sharing information with others. Islands of experimentation are common in organizations that do not conform to holistic AI organizations. An AI Center of Excellence remedies isolation by connecting isolated islands, including teams and people. The AI Center of Excellence is the mental muscle of your organization. It acts as a center to accept and distribute work, and it also logs what has been done and how accomplishments were achieved (including failures) by listing people, skills, and tools, so that new projects can leverage previous experience and new technological advancements. Large organizations such as Oracle have many AI Centers of Excellence. For example, the Infrastructure business has an AI Center of Excellence, which I founded and managed. Oracle Healthcare has another AI Center of Excellence.

Finally, this chapter also covered the major pillars of the AI Center of Excellence: Best Research Practices, Human Centered AI, Data, and MLOps.

Chapter 4: AI Innovation [This content is currently in development.]

This content is currently in development.

Chapter 5: AI Product Management [This content is currently in development.]

This content is currently in development.

Chapter 6: AI Research [This content is currently in development.]

This content is currently in development.

Chapter 7: AI Deployment [This content is currently in development.]

This content is currently in development.

Chapter 8: AI Production [This content is currently in development.]

This content is currently in development.