Introduction

Summary
Chapter 1: Strategy
How to:
• identify and prioritise problems for AI to solve
• evaluate AI deployment strategies – from third-party APIs to in-house teams
• plan budgets and timescales for AI projects
• build buy-in for AI and mitigate culture concerns

Chapter 2: People
How to:
• understand the different roles in an AI team
• structure an AI team according to your objectives
• source, evaluate, attract and retain AI talent

Chapter 3: Data
How to:
• develop a data strategy for AI
• accelerate data acquisition
• structure, secure and provide data
• develop a high-quality data set
• understand and minimise bias

Chapter 4: Development
How to:
• understand the advantages and limitations of different development approaches
• select a development strategy
• choose hardware for AI development
• address problem domains with suitable AI techniques
• evaluate the strengths and limitations of popular deep learning frameworks
3

4
12



22



32
44
Contents
Explore our companion report
The State of AI 2019: Divergence
to understand AI today, what’s to
come and how to take advantage.
MMC Ventures
MMC Ventures is a research-led venture capital firm that has backed over 60 early-stage, high-growth
technology companies since 2000.
MMC’s dedicated research team provides the Firm with a deep and differentiated understanding
of emerging technologies and sector dynamics to identify attractive investment opportunities.
MMC’s research team also supports portfolio companies through the life of MMC’s investment.
MMC helps to catalyse the growth of enterprise software and consumer internet companies that have the
potential to disrupt large markets. The Firm has one of the largest software-as-a-service (SaaS) portfolios
in Europe, with recent exits including CloudSense, Invenias and NewVoiceMedia. MMC’s dynamic
consumer portfolio includes Bloom & Wild, Gousto and Interactive Investor.
MMC Ventures Research
David Kelnar – Partner & Head of Research
Asen Kostadinov, CFA – Research Manager
Explore MMC’s cutting-edge research at mmcventures.com, MMC Writes (www.medium.com/mmc-writes)
and @MMC_Ventures on twitter.
www.mmcventures.com
@MMC_Ventures
Barclays UK Ventures
Barclays UK (BUK) Ventures is a specialist business unit within Barclays with an independent mandate to
deliver new customer experiences at pace and scale – driving growth for communities, businesses and
Barclays. BUK Ventures identifies, incubates and scales transformational new business lines and business
models both within and outside of Barclays through organic build-out, commercial partnerships and
venture investments.
BUK Ventures comprises a strong team of intrapreneurs, including Eagle Labs, who work to create thriving
communities with the aim of connecting businesses of all sizes to the networks they need to succeed.
www.home.barclays
1
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Introduction
Your blueprint for AI
Artificial Intelligence (AI) is today’s most important enabling technology.
Leading startups, scale-ups and enterprises are using AI today to reimagine
consumer experiences and business processes – unlocking revenue growth
and cost savings at the expense of their competitors. How can you take advantage?
Embracing AI can seem daunting. How do I develop an AI strategy? Can I afford AI?
Do I need an in-house team? Which algorithms should I explore?
We’re excited to offer a blueprint for success. Our AI Playbook is an accessible, step-
by-step guide to taking advantage of AI. We explain, without jargon, best practice
in six core competencies for AI: strategy; people; data; development; production;
and regulation & ethics. In a hurry? Every chapter includes a one-page summary
and checklist of actions.
Whether you: are a founder, executive, information worker or developer; work at a
startup, scale-up or enterprise; and have a budget of £500 or £5m – this Playbook
will catalyse your journey.
Our research has been enriched by 400 discussions with Europe’s leading AI startups,
scale-ups and practitioners. Our special thanks to Dr. Janet Bastiman, MMC Ventures’
AI Venture Partner, for her invaluable expertise.
At MMC Ventures we invest in, and support, the UK’s most promising entrepreneurs.
If you’re an early stage company, get in touch to see how we can accelerate your journey.
David Kelnar
Partner & Head of Research
MMC Ventures
Email: david.kelnar@mmcventures.com
Twitter: @davidkelnar
Chapter 5: Production
How to:
• optimise your research and development activity
• select a hosting environment
• transition development systems to live use
• measure and monitor system accuracy
• develop a robust quality assurance process
• implement an effective maintenance programme

Chapter 6: Regulation & Ethics
How to:
• comply with GDPR data handling requirements
• verify that automated systems meet regulatory stipulations
• explore different approaches to ‘explainability’
• apply a framework for ethical data use
We’d value your feedback so we can improve the Playbook.
Get in touch at insights@mmcventures.com
66








76
Contents
Leading startups,
scale-ups and enterprises
are using AI today to
reimagine consumer
experiences and
business processes.
Our AI Playbook is an
accessible, step-by-step
guide to taking advantage
of AI. We explain, without
jargon, best practice
in six core competencies.
If you’re an early stage
company, get in touch
to see how we can
accelerate your journey.
3
2
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Introduction
Summary
Chapter 1: Strategy
• Recognise AI’s potential for value creation. While you
should not add AI to your initiatives for the sake of doing so,
you risk losing competitive advantage if you fail to explore
what AI can offer.
• Identify appropriate problems for AI to solve. AI is
particularly effective at: assignment (identifying what
something is, or the extent to which items are connected);
grouping (determining correlations and subsets in data);
generation (creating images or text based on inputs) and
forecasting (predicting changes in time series data).
All businesses will have challenges where the above
apply and, therefore, where AI can be fruitful.
• Prioritise projects according to value and viability. Ensure
you have a clear, concise specification of the problem and
desired outcomes. Assessing viability includes considering
whether your training data is balanced (free from bias),
exhaustive (captures all relevant variables), diverse (captures
rare situations) and is of sufficient volume.
• Timescales for creating AI are less certain than for traditional
software development – and typically extend non-linearly
with desired accuracy. Timescales vary according to
the problem type, subject domain and data availability.
Frequently, a prototype with limited accuracy can be
developed within three months.
• Align your budget with your goals and deployment
strategy. The budget an AI initiative requires will depend
on multiple factors including the complexity of the
problem, the availability and quality of training data, and the
deployment strategy you select.







• AI deployment strategies include: calling third party
Application Programming Interfaces (APIs); using managed
AI services from third parties; building a small in-house AI
team; and building an extensive in-house AI team. A large,
in-house team is a multi-million-pound annual investment.
Many companies develop a proof-of-concept using their
existing development teams, and third-party APIs or paid
services. Then, they create a budget proposal and begin
with a small, in-house AI team.
• Seek sponsorship from senior executives. Support from
management will be important for new AI initiatives to
succeed. To build support, educate senior management
regarding the benefits of AI while setting realistic
expectations regarding timescales and results.
• Anticipate and mitigate cultural concerns about AI.
To some, AI will be unfamiliar. Others will see their
workflows change. Many people may be concerned about
the impact of AI on job security. Frequently, AI will enhance
an individual’s role by offering ‘augmented intelligence’.
Address concerns proactively by highlighting the ways in
which AI will support individuals’ goals and enable team
members to redirect their time to engaging aspects of
their roles.
• Expect non-traditional security considerations. Protect
against malicious activity via thorough system testing and
exception handling.
• When your first project is underway, anticipate the longer-
term aspects of your AI strategy. Consider: maintenance;
data (budget to retrain your system as data evolves and
increases); evolving algorithms (new techniques will offer
better results in the future); scaling (extending useful AI
systems to additional business units and geographies);
innovation (a roadmap for new AI initiatives); and regulation
(a strategy to comply with new legislation as it emerges).
Chapter 2: People
• In AI, job titles vary and can be difficult to interpret.
We describe characteristics and salaries for six key roles:
Data/Machine Learning Engineer; Data scientist; Machine
Learning Researcher; Head of Data; Head of Research/
AI; and Chief Scientist/Chief Science Officer. For each,
individuals’ capabilities vary across competencies in
research, engineering, production and strategy.
• The composition of your team should depend upon the
problem being solved and your approach to doing so. It is
advisable, however, to avoid hiring solo talent. Begin with
a small team, and ensure you have a robust AI strategy in
place before expanding your AI personnel.
• We suggest team structures, first hires and next steps for
six scenarios: “I want insights into internal data”; “I want to
implement third party AI APIs”; “I want to outsource AI



development”; I want to create bespoke AI models”;
“I want to use a combination of bespoke and third party AI”;
and “I have an idea that’s cutting edge.”
• Recruiters, conferences and universities are primary sources
of talent. Traditional recruitment agents find it difficult to
screen AI candidates, so engage with specialist recruiters.
Conferences and meetups are powerful vehicles for talent
acquisition; be active in the AI community, attend and
speak at conferences, and grow your network to discover
capable candidates. Engage with universities; post on their
job boards, establish partnerships and pay for projects to
engage students who may seek future opportunities with you.
• Diversity delivers economic value and competitive
advantage. Review the culture in your company, AI team
and hiring practices to ensure diversity, representation
and inclusion.
Myth
“AI is a distant dream.”
“We don’t have the budget
to implement AI.”
“AI is dominated by the big technology
companies. There’s no point in my
company trying to compete.”
“We can’t use AI because our business
requires explainable processes.”
“I can throw AI at my data and it
will offer efficiencies.”
While general, human-level artificial intelligence will not be available for
many years, there are many applications for AI that are viable today and offer
companies cost savings and revenue growth.
While a large, in-house AI team will require extensive investment, third parties
offer access to AI services (via API) for as little as several hundred pounds.
Further, as AI democratises, growing libraries of pre-trained models offer
results at low cost. If you have a software engineering team, you can validate
benefit from AI at minimal cost.
While companies including Amazon, Google, IBM and Microsoft have developed
extensive AI services, they lack the strategic desire, data advantage or domain
expertise to tackle the many sector- or function-specific applications for AI.
Today, a rich ecosystem of startups, scale-ups and corporates are deploying
AI for competitive advantage.
There are several ways to explain what is occurring inside an AI system
(see Chapter 6). Some AI is directly explainable. With deep learning systems,
where explainability is a challenge, it is possible to explain how input variables
influence output.
AI is a tool that requires a structured problem and appropriate data to
be effective.
Reality
To engage effectively with AI, separate AI myths from reality
Source: MMC Ventures
5
4
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Summary
• An effective job description should emphasise projects
(the nature of the engagements on which the successful
candidate will work), skills and impact. Most data scientists
seek work that will ‘make a difference’. To attract talent,
demonstrate how the successful candidate’s work will do so.
• When hiring, prioritise adaptable problem-solvers.
In addition to having role-specific and technical skills,
a strong AI candidate will: understand available tools to
enable rapid research and development; appreciate when
to release an imperfect solution and when to wait; and
communicate and collaborate well.
• Optimise every stage of your recruitment funnel.
We provide best practices for: CV screening; phone
screening; technical testing; face-to-face interviews and
post-interview follow-up.
• AI talent is in short supply. Challenge, culture and company
are key for retention. In addition to an attractive financial
package, consider: offering flexible working hours; offering
challenging problems and minimising drudgery through
automation; creating a culture in which diverse ideas are
shared; avoiding ‘lone workers’; ensuring your AI team
receives recognition for its work; and supporting team
members’ publishing and presentation of work.
Chapter 3: Data
• For effective AI, develop a data strategy. A data strategy
spans: data acquisition & processing; quality; context;
storage; provisioning; and management & security.
Define your data strategy at the outset of your AI initiative.
• Accelerate data acquisition by using multiple sources.
Developers draw on several sources including: free
resources (such as dataset aggregators); partnerships with
third parties (companies, universities, data providers and
government departments); and new, proprietary data.
• A high-quality data set has appropriate characteristics to
address your business challenge, minimises bias and offers
training data labelled with a high degree of accuracy.
Develop a balanced data set - if you possess significantly
more samples of one type of output than another, your
system will exhibit bias.
• Primary forms of bias are: unwarranted correlations
(between inputs and output classifications); erroneous
assumptions which cause relationships to be missed
(‘underfitting’); and modelling noise instead of valid outputs
(‘overfitting’). Adjust for overfitting and underfitting by




using different data volumes and model structures.
Remove unwarranted correlations through testing.
• Ensure that the results of your internal testing will be
maintained when applied to real-world data. Test early,
and frequently, on real-world data.
• Managing ‘dirty data’ is data scientists’ most significant
challenge (Kaggle). Smaller volumes of relevant, well-
labelled data will typically enable better model accuracy
than large volumes of poor-quality data. To label data
effectively: consider developing a supporting system to
accelerate data labelling and improve accuracy; draw on
existing AI and data techniques; and seek data labelled by
multiple individuals to mitigate mislabelling.
• Understand the data you use. Ensure you capture the
human knowledge regarding how your data was gathered,
so you can make downstream decisions regarding its use.
Capture data provenance (where your data originated and
how it was collected). Define your variables (differentiate
between raw data, merged data, labels and inferences).
Understand the systems and mappings through which
your data pass to retain detail.
Chapter 4: Development
• There are many ways your company can engage with AI.
Use third party AI APIs; outsource; use a managed service;
build an in-house team; or adopt a ‘hybrid’ approach
combining an in-house team with third party resources.
• Third party AI APIs fulfil specific functions to a moderate
or high standard at low cost. Most solve problems in
the domains of vision and language. Numerous APIs
are available from Amazon, Google, IBM, Microsoft
and also other smaller companies. Features vary; we
provide a summary. APIs offer immediate results without
upfront investment, at the expense of configurability and
differentiation. Use an API if you seek a solution to a generic
problem for which an API is available. APIs are unsuitable
if you seek solutions to narrow, domain-specific problems,
wish to configure your AI, or seek long-term differentiation
through AI.
• Managed services enable you to upload your data,
configure and train models using a simple interface, and
refine the results. Managed services abstract away much of
the difficulty of developing AI and enable you to develop a
custom solution rapidly. Managed services offer greater


flexibility and control than APIs, but less flexibility than an
in-house team, and also require you to transfer data to a
third party and may create dependencies.
• If a third-party solution is unavailable and an in-house team
is too expensive, you can outsource your AI development.
Whether outsourcing is appropriate will depend upon
your domain, expertise, required time to value and data
sensitivity. If outsourcing, specify desired frameworks
and standards, who will provide training data, costs,
timescales and deployment considerations. Outsource if
you require trusted expertise quickly and a cheaper option
than permanent employees. Avoid outsourcing if your
data permissions prohibit it, you require domain or sector
knowledge that an outsourcer lacks, or you wish to build
knowledge within your own company.
• An in-house AI team offers maximum control, capability
and competitive differentiation – at a price. A small in-
house team will cost at least £250,000 to £500,000 per
year. A large team requires a multi-million-pound annual
investment. To develop an in-house team your company
must also: attract, manage and retain AI talent; select
• Store and structure data optimally to support your
objectives. Storage options include basic file-based,
relational, NoSQL or a combination. When selecting
storage plan for growth in data volume, updates,
resilience and recoverability.
• One in three data scientists report that access to data is
a primary inhibitor of productivity (Kaggle). Develop a
provisioning strategy that: ensures data is accessible across
your organisation when needed; contains safeguards to
protect your company against accidents; optimises system
input/output; and maintains data freshness.
• Implement robust data management and security
procedures consistent with local and global regulations.
Personal data is protected by UK and EU law and you must
store it securely. Draw on principles of appropriate storage,
transmission and minimum required access.
The six components of an effective data strategy
Source: MMC Ventures
Fig. X: The six components of an effective data strategy
Ma
nag
ement & Security
Context
Provisioning
Storage
Quality
Acquisition
& Processing
Most data scientists seek work that
will ‘make a difference’. To attract
talent, demonstrate how the successful
candidate’s work will do so.
7
6
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Summary
development frameworks and techniques; gather and
cleanse data; learn how to productise AI into real-world
systems; and comply with regulatory and ethical standards.
Build an in-house team if you have a problem that cannot
be solved with existing solutions, seek differentiation in the
market, or seek to maintain control over your data.
• A ‘hybrid’ approach is ideal for many companies. Plan for
an in-house team that will address your requirements to a
high standard over time, but use third party APIs to solve an
initial, simpler version of your challenge. A hybrid approach
can be attractive if you seek rapid initial results, wish to
limit spend until a business case is proven and want greater
differentiation and resilience over time.
• To develop AI yourself you have choices to make regarding
your AI ‘technology stack’. The stack comprises six layers:
hardware; operating systems; programming languages;
libraries; frameworks; and abstractions. Not all problems
require the full stack.
• Ensure your team has hardware with graphical processing
units (GPUs) that support NVIDIA’s CUDA libraries.
Laptops with high performance graphics cards offer
flexibility. For greater power, desktop machines with
powerful GPUs are preferable. To train large models,
use dedicated servers. Cloud-based servers offered by
Amazon, Google or Microsoft are suitable for most early
stage companies.
• Apply AI techniques suited to your problem domain.
For assignment problems consider: Support Vector
Classification; Naïve Bayes; K-Nearest Neighbour
Classification; Convolutional Neural Networks; Support
Vector Regression; or ‘Lasso’ techniques. We describe each
and explain their advantages and limitations. For grouping
problems, explore: Meanshift Clustering; K-Means; and
Gaussian Mixture Models. For generation, consider:
Probabilistic Prediction; Variational Auto-Encoders; and
Generative Adversarial Networks (GANs).
Source: https://medium.freecodecamp.org/an-intuitive-introduction-to-generative-adversarial-networks-gans-7a2264a81394
With one network, GANs generate output from random noise; a second network serves as a discriminator
DISCRIMINATOR
GENERATOR
Training set
Random noise
Fake image
Real
Fake
Fig. X: With one network, GANs generate output from random noise;
GANs use a secod network as a discriminator
Chapter 5: Production
• An unused AI system delivers no value. Develop a
production process that smoothly transitions AI systems
you have in development to live use.
• AI production follows a conventional development
process and requires you to undertake research, develop
a prototype and create a minimum viable product (MVP).
Once in production, undertake cycles of ideation, research,
development and quality assurance.
• Effective R&D requires rapid iteration. Initially, optimise
for speed over quality. Releasing an early model into
production for feedback is preferable to waiting until a
research model is perfect.
• During the R&D phase, solicit feedback about prototypes
from beyond the AI and production teams to minimise
expensive redevelopment later.



• When moving from MVP to production, select an
appropriate hosting environment. On-premise hosting is
suitable for those with highly sensitive data and existing
on-premise hardware, but is rarely preferred by early stage
companies given high upfront costs, unpredictable activity
levels and required security expertise. Hosting your own
hardware in a data centre offers control and value over
the long term. Upfront costs can be high, however, and
managing a data centre can prove a distraction for young
companies. Cloud hosting, which offers low upfront costs
and high levels of flexibility, is well suited to many early
stage companies – although annual costs can be double
that of a self-managed data centre and cloud hosting
may be unsuitable for highly sensitive data. Consider the
physical location in which your cloud servers are hosted.
Different countries have varying rules regarding data
and you may be required to keep your data within its
area of origin.
• Proving that AI systems are effective differs from the typical
software quality assurance (QA) process. Test your AI
system at multiple stages – during training, validation and
continuously through its life. Efficiency is critical; automate
testing to as great an extent as possible.
Source: MMC Ventures
The AI production pipeline is similar to a normal development practice
IDEA
QA
RESEARCH
PRODUCTION
MVP
Fig. X: Title TBC
R&D
IDEAS
FEATURES
PROTOTYPE
9
8
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Summary
• Understand the three common measures of ‘accuracy’ in
AI – recall, precision and accuracy – and monitor all three
to capture performance. Balancing precision and recall is
challenging. Whether you elect to minimise false positives
or false negatives should depend upon the nature of your
sector and the problem you are solving.
• An effective maintenance programme will sustain your AI’s
intelligence. Beyond the maintenance you would typically
perform on a software system, you should verify and
update your AI system on an ongoing basis. AI technology
is developing at pace. Invest in continual improvement to
ensure your system avoids obsolescence.
Chapter 6: Regulation & Ethics
• As consideration of data privacy grows, and with the
General Data Protection Regulation (GDPR) in force across
the European Union (EU), it is vital to ensure you are using
data appropriately. The GDPR applies to all companies
processing the personal data of people in the EU,
regardless of a company’s location.
• Companies that are ‘controllers’ or ‘processors’ of personal
information are accountable for their handling of individuals’
personal information. Demonstrate compliance with
GDPR data handling requirements and the principles of
protection, fairness and transparency.
• Minimise the personal data you require, to reduce
regulatory risk, and pseudonymise all personal data through
anonymisation, encryption or tokenisation.
• In addition to standardising data handling requirements and
penalties for misuse, the GDPR introduced considerations
that can impact AI systems specifically. Verify that
automated systems meet GDPR stipulations. Article 22
of the GDPR prohibits legal effects that result solely from
automated processing being undertaken without an


individual’s explicit consent, when consent is required.
Several legislative terms are subject to interpretation at this
time. It may be prudent to make your system advisory only,
and include a human check, if you are developing a system
that could materially impact an individual’s life.
• ‘Explainability’ – explaining how the outputs of your AI
system are derived – is growing in importance. Convention
108 of the Council of Europe, adopted into UK and EU law
in May 2018, provides individuals with the right to obtain
knowledge of the reasoning underlying data processing
systems applied to them. Explainability can be challenging
in relation to deep learning systems. Explore varying
approaches to explainability including Inferred Explanation,
Feature Extrapolation and Key Variable Analysis. Each offers
trade-offs regarding difficulty, speed and explanatory power.
• Develop a framework for ethical data use to avoid
reputational and financial costs. The ALGOCARE
framework, developed by the Durham Police Constabulary
in partnership with academics, highlights issues you should
consider when managing data. It incorporates: the nature
of system output (Advisory); whether data is gathered
lawfully (Lawful); whether you understand the meaning of
the data you use (Granularity); who owns the intellectual
property (IP) associated with the data (Ownership); whether
the outcomes of your system need to be available for
individuals to challenge (Challenge); how your system
is tested (Accuracy); whether ethical considerations are
deliberated and stated (Responsible); and whether your
model has been explained accessibly to as great an extent
as possible (Explainable).
Companies that are ‘controllers’ or
‘processors’ of personal information
are accountable for their handling
of individuals’ personal information.
Demonstrate compliance with GDPR data
handling requirements and the principles
of protection, fairness and transparency.
Source: MMC Ventures
How to select an approach to explainability
Use this approach if you:
Inferred Explanation
Feature Extraction
Key Variable Analysis
– Seek a high-level overview of your AI system
– Believe correlation offers sufficient explainability
– Require detail from within the network
– Have a network type (e.g. images) where abstractions can
be mapped onto input data
– Require detail about the importance of variables
– Seek to prevent unwanted bias in your variables
– Require detail regarding how variables lead to decisions
– Have limited time
– Require precise impact of input variables, not general
features
– Are not using an assignment–based or generative
AI network
– Have limited time
– Seek to the publish your results
– Wish to offer a layperson’s guide to your model
Avoid this approach if you:
Understand the three common
measures of ‘accuracy’ in AI –
recall, precision and accuracy
– and monitor all three to
capture performance.
11
10
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Summary
Strategy
Chapter 1
Summary
• Recognise AI’s potential for value creation. While you
should not add AI to your initiatives for the sake of doing so,
you risk losing competitive advantage if you fail to explore
what AI can offer.
• Identify appropriate problems for AI to solve. AI is
particularly effective at: assignment (identifying what
something is, or the extent to which items are connected);
grouping (determining correlations and subsets in data);
generation (creating images or text based on inputs) and
forecasting (predicting changes in time series data).
All businesses will have challenges where the above
apply and, therefore, where AI can be fruitful.
• Prioritise projects according to value and viability. Ensure
you have a clear, concise specification of the problem and
desired outcomes. Assessing viability includes considering
whether your training data is balanced (free from bias),
exhaustive (captures all relevant variables), diverse (captures
rare situations) and is of sufficient volume.
• Timescales for creating AI are less certain than for traditional
software development – and typically extend non-linearly
with desired accuracy. Timescales vary according to
the problem type, subject domain and data availability.
Frequently, a prototype with limited accuracy can be
developed within three months.
• Align your budget with your goals and deployment
strategy. The budget an AI initiative requires will depend
on multiple factors including the complexity of the
problem, the availability and quality of training data, and the
deployment strategy you select.




• AI deployment strategies include: calling third party AI APIs;
using managed AI services from third parties; building a
small in-house AI team; and building an extensive in-house
AI team. A large, in-house team is a multi-million-pound
annual investment. Many companies develop a proof-
of-concept using their existing development teams, and
third-party APIs or paid services. Then, they create a budget
proposal and begin with a small, in-house AI team.
• Seek sponsorship from senior executives. Support from
management will be important for new AI initiatives to
succeed. To build support, educate senior management
regarding the benefits of AI while setting realistic
expectations regarding timescales and results.
• Anticipate and mitigate cultural concerns about AI.
To some, AI will be unfamiliar. Others will see their
workflows change. Many people may be concerned about
the impact of AI on job security. Frequently, AI will enhance
an individual’s role by offering ‘augmented intelligence’.
Address concerns proactively by highlighting the ways in
which AI will support individuals’ goals and enable team
members to redirect their time to engaging aspects of
their roles.
• Expect non-traditional security considerations. Protect
against malicious activity via thorough system testing and
exception handling.
• When your first project is underway, anticipate the longer-
term aspects of your AI strategy. Consider: maintenance;
data (budget to retrain your system as data evolves and
increases); evolving algorithms (new techniques will offer
better results in the future); scaling (extending useful AI
systems to additional business units and geographies);
innovation (a roadmap for new AI initiatives); and regulation
(a strategy to comply with new legislation as it emerges).
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
12
13
Chapter 1
Strategy
AI is a powerful tool. Before you invest time and money in the
technology, you need a strategy to guide its use. Without an AI
strategy, AI will become an additional cost that fails to deliver a return
on investment. Below, we describe how to: identify appropriate
use cases for AI; select your first AI initiative; explore deployment
strategies; anticipate timescales; predict required budget; and
establish the cultural buy-in necessary for success.
Recognise AI’s potential for value creation
AI is a powerful set of techniques offering companies tangible
cost savings and increased revenue. Further, adoption of
AI is ‘crossing the chasm’, from innovators and early adopters
to the early mainstream. While you should not attempt to add
AI to your initiatives for the sake of doing so, and should be
mindful of its limitations, you risk losing competitive advantage
if you fail to explore what AI can offer. Approach AI based on
its transformational potential.
To engage effectively with AI, separate AI myths from reality:
Myth
“AI is a distant dream.”
“We don’t have the budget
to implement AI.”
“AI is dominated by the big technology
companies. There’s no point in my
company trying to compete.”
“We can’t use AI because our business
requires explainable processes.”
“I can throw AI at my data and it
will offer efficiencies.”
While general, human-level artificial intelligence will not be available for
many years, there are many applications for AI that are viable today and offer
companies cost savings and revenue growth.
While a large, in-house AI team will require extensive investment, third parties
offer access to AI services (via API) for as little as several hundred pounds.
Further, as AI democratises, growing libraries of pre-trained models offer
results at low cost. If you have a software engineering team, you can validate
benefit from AI at minimal cost.
While companies including Amazon, Google, IBM and Microsoft have developed
extensive AI services, they lack the strategic desire, data advantage or domain
expertise to tackle the many sector- or function-specific applications for AI.
Today, a rich ecosystem of startups, scale-ups and corporates are deploying
AI for competitive advantage.
There are several ways to explain what is occurring inside an AI system (see
Chapter 6). Some AI is directly explainable. With deep learning systems,
where explainability is a challenge, it is possible to explain how input variables
influence output.
AI is a tool that requires a structured problem and appropriate data to
be effective.
Reality
Identify use cases
Prioritise initiatives
Build buy-in
Mitigate cultural and security concerns
Develop a long-term strategy
Understand timescales
Develop a budget
Strategy: The Checklist

Understand the categories of problem AI can address


Seek ideas and advice from AI practitioners


Create a list of potential AI initiatives offering business benefit

Develop a clear statement of the business challenge and opportunity

Define measures of success


Review the suitability of available data

Define the return on investment of your AI strategy


Develop a clear, detailed implementation plan


Educate senior management regarding AI and establish realistic expectations


Educate and involve your workforce to address concerns


Plan for non-traditional security challenges


Set aside budget for maintenance, updates and re-training


Review new technology and evolving data sets

Develop plans to extend your AI to additional business units and to undertake new AI initiatives


Develop a process to ensure compliance with evolving legislation

Appreciate the need to iterate AI systems


Develop realistic goals regarding accuracy and timescales

Understand budget requirements for different AI development strategies


Select an initial and long-term development strategy for creating AI systems
Fig. 1. Separate AI myths from reality
Source: MMC Ventures
15
14
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 1
Strategy
Identify appropriate problems
AI can be effective at solving problems – but it is important
to begin with a clear problem in mind. Broad considerations
are insufficient. When creating a list of potential AI initiatives,
develop a precise definition of a problem you wish to address.
“Always focus on the problem you’re using AI to solve”
(Tim Sadler, Tessian). Do you have a problem whose solution
will add value within the business or to customers? Can the
problem be solved using AI? AI is particularly effective in
four problem domains: assignment; grouping; generation
and forecasting (Fig. 2).
There are many ways to identify and evaluate potential AI
projects, including:
• Network: to familiarise yourself with AI and its use cases,
engage with your professional network and AI communities
on LinkedIn and Meetup.com (on Meetup.com,
communities can establish informal gatherings and there
is a thriving AI community). Many community events are
free. Ask an attendee for a coffee and you will find a useful
sounding board for your questions and ideas. Informal
advice is valuable; you can discuss whether AI might be
suited to your use cases, why, and how to turn your idea
into an initiative. “Find someone who is already using AI
and bounce your ideas off them. Work out if your idea is
possible. Have that conversation before even thinking
about a consultant.” (Miguel Martinez, Chief Data
Scientist, Signal).
• Conferences: seek inspiration, talk with experts and
understand industry best practises through conferences.
Conferences tend to be high-level executive briefings, sales
pitches or presentations of academic research. If you are
early in your AI journey, prioritise events with multiple tracks
for less experienced practitioners, or a mixture of levels so
you receive an overview. Useful conferences will provide
access to companies with successful AI solutions, which
you can talk to for advice and collaboration. The cost of
conference attendance varies from several hundred pounds
to several thousand. Familiarise yourself with sessions
before you book to ensure a return on investment.
“Always focus on the problem
you’re using AI to solve.”
Tim Sadler, Tessian
Source: MMC Ventures
Fig. 2. AI is highly effective at Assignment, Grouping, Generation and Forecasting
Problem Domain
Definition
Assignment
Grouping
Generation
Forecasting
• Identify what something is
(classification)
• Identify how connected
items are (regression)
• Given data, determine
correlations and subsets
(clustering)
• Given an input, create an
image or text (generation)
• Given time series data,
predict future changes
(sequencing)
• Understand the sentiment of text
• Recognise logos in images
• Make a medical diagnosis based on symptoms
• Quantify the relationship between a preservative and product
shelf life
• Evaluate how consumer income affects propensity for impulse
purchasing
• Predict the purchase price of a second-hand vehicle based
upon its condition
• Identify social subgroups within a customer base for
enhanced targeting
• Evaluate factors that correlate with patient melanomas
• Identify themes in customer feedback surveys
• Create a chatbot for customer service
• Translate customer conversations to a different language
• Create photorealistic media for advertising
• Predict weekly sales to avoid the loss of perishable stock
• Determine the probability of equipment failure to enable
proactive replacement
• Predict exchange rate fluctuations
Examples
Sector
Asset Management
Healthcare
Insurance
Law & Compliance
Manufacturing
Retail
Transport
Utilities
Investment strategy
Diagnostics
Risk assessment
Case law review
Predictive
maintenance
Customer
segmentation
Autonomous
vehicles
Supply management
Portfolio construction
Drug discovery
Claims processing
Due diligence
Asset performance
optimisation
Content
personalisation
Infrastructure
optimisation
Demand optimisation
Risk management
Patient monitoring
Fraud detection
Litigation strategy
Utility optimisation
Price optimisation
Fleet management
Security
Client service
Customer service
Surgical support
Compliance
Supply chain
optimisation
Churn prediction
Control applications
Customer experience
Example use cases
All businesses will have challenges of the types above –
and therefore problems to which AI can be usefully applied.
The table below provides examples of popular AI use cases.
Fig. 3. AI is being fruitfully applied to a wide variety of use cases
“The applications of AI are endless.”
Timo Boldt, Gousto
Source: MMC Ventures
17
16
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 1
Strategy
Prioritise projects according to value
and viability
Once you have ideas for AI projects, beyond assessing the
relative value of each to your company, determine the most
viable by addressing the following questions. As well as
enabling you to choose a feasible project, the answers will
help you define project parameters and objectives.
• Problem: Does the project fall within the definition of
assignment, grouping, generation or forecasting? If you
cannot clearly define the type of problem, it may be a viable
undertaking but is unlikely to be an ideal first AI project for
your company.
• Definition: Can you state the problem clearly and
concisely? If not, you will lack a clear definition of the
system’s purpose and will struggle to select and employ
appropriate AI techniques.
• Outcomes: Can you define the levels of accuracy and
speed the system must achieve to be successful? Avoid
initiatives that lack these measures. If converting an existing
manual process, do you know the accuracy and speed
of the current workflow? If you are undertaking a new
initiative for your company, define what will be deemed
a successful outcome.
• Data: Do you have sufficient data to train and test a system?
Without adequate, high quality data to train your system
your initiative will fail. If you are choosing between a range
of otherwise viable projects, select the engagement
supported by the greatest quantity of high-quality data.
It can be challenging to assess data suitability. Typically, data
must be:
• Representative: Data you use to train your AI model
should reflect the data you will feed your system in its live
environment. If the data differs significantly, results will be
poor even if the accuracy of your system during training
is high.
• Diverse: Even rare situations should be captured in
available training data. Without diverse data, your system
may not generalise effectively. Overall accuracy may be
high, but your model will fail (misclassify, wrongly correlate
or poorly predict) in less frequent situations.
• Balanced: A biased data set produces a biased system.
Does your data have inherent bias? For example, are you
analysing CVs for suitability to a role and most candidates
are of the same gender? Liaise with individuals in your
organisation who understand your data and can advise on
its inherent bias.
• Exhaustive: All relevant variables must be included in the
available data. For assignment and grouping problems,
missing variables will lead to oversimplified results
(unwarranted correlations). In other problem domains,
you may be unable to derive utility from your system.
• Sufficient: While a smaller volume of high-quality data
is preferable to extensive, poor-quality data, the volume
of data you can acquire must be sufficient to train your
algorithm well. For assignment problems, useful results
frequently begin to emerge after approximately 1,000
examples for each output label. Some problems require
more or fewer. For forecasting problems, you may require
data spanning at least double the duration of the periodicity
of the item forecasted. For grouping and generation
challenges, typically output improves with data volume
but again 1,000 examples are frequently a minimum.
Typically, the more complex the challenge, the more data
points you will require.
In Chapter 3, we explain how to develop a full data strategy
to support your AI initiatives.
Timescales will extend non-linearly
with accuracy
Timescales for AI initiatives can be less certain than for
traditional software development. AI systems cannot
predictably be developed once, tested and then deployed.
Typically, multiple cycles of training are required to identify
a suitable combination of data, network architecture and
‘hyperparameters’ (the variables that define how a system
learns). These dynamics will vary according to domain,
the nature of the problem and the data available.
Accordingly, it can be challenging to predict or automate
AI initiatives unless they are similar to projects you have
previously undertaken.
While timescales will vary according to the problem you are
addressing, the resources you have committed and the buy-in
you have achieved, you can frequently develop a prototype
within three months. It may take days to develop a first version
of a system that offers 50% accuracy, weeks to improve the
system to 80% accuracy, months to achieve 95% and much
longer for additional incremental improvements (Fig. 4).
For straightforward problems, expect a similar progression but
over shorter timescales. For particularly challenging problems,
which require extensive data to describe the problem or new
techniques to solve it, this timeline may extend significantly.
“Solving really hard problems using AI takes time and depth. It
follows a different curve” (Fabio Kuhn, Vortexa).
Align your budget with your goals and
deployment strategy
The budget you require for your AI initiatives will depend upon
multiple factors including:
• the nature, complexity and domain-specificity of the
projects you undertake;
• available, and preferred, development strategies (use of
third-party services versus an in-house development team);
• availability, quality and consistency of relevant data;
• a well-considered starting point;
• regulatory and ethical considerations to be addressed.
Some challenges can be addressed with a readily-available
third-party application programming interface (API). Others
may be solved with a single pass of data through an existing,
public domain network architecture. Others still will require
extensive research and multiple iterations of training and
adjustment to meet success conditions. Costs will vary
according to the development strategy you select.
The following strategies offer progressively greater
functionality and uniqueness in return for increased spend:
• Third-party APIs: If another company has already solved
your business problem, and you need only call the
counterparty’s service via an API to receive a result,
prices can start as low as several hundred pounds.
Using third-party APIs is the fastest way to deploy AI in
your company and requires minimal time from your existing
development team.
• Bespoke third-party services: To obviate the need for your
own AI team, you can engage third-parties to develop and
train your AI models. You will need to gather and prepare
your own data and have a broad overview of the process
of creating models. You are unlikely to require a budget of
more than a few thousand pounds for training and running
costs, plus the cost of an individual – ideally a data expert
already in your business – with an understanding of AI to
manage the process.

Without adequate, high quality
data to train your system, your
initiative will fail.
Costs will vary according
to the development strategy
you select.
“Solving really hard problems using
AI takes time and depth. It follows a
different curve. Endurance is key.”
Fabio Kuhn, Vortexa
Source: MMC Ventures
Fig. 4. Timescales typically increase non-linearly with
desired accuracy
0%
25%
50%
75%
100%
Time
AccuracyFig. 6: Timescales typically increase non-linearly with desired accuracy
19
18
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 1
Strategy
• A small, in-house team: A dedicated in-house AI team is
likely to cost at least £250,000 to £500,000 per year, even
for a small team. Whether you seek to repurpose publicly-
available models, or solve unique problems, you will need
to pay for: two to four individuals; the hardware they require
to train and run their models and potentially extra hires for
productionising the resulting system.
• A large, in-house team: An extensive team, recruited to
solve problems at the edge of research, will require a multi-
million-pound investment in personnel and hardware.
This investment may yield a unique AI offering. It should
only be considered as a first step, however, if your challenge
cannot be solved with existing AI techniques and solutions,
if you have access to unique data, and if you face significant
restrictions on your ability to pass data to third parties.
We describe, in detail, the advantages and disadvantages of
different development strategies in Chapter 4 (Development).
You may wish to develop a proof-of-concept, using your
existing development team and third-party APIs or paid
services, before creating a budgetary proposal. Most
companies then start with the small, dedicated AI team.
Seek sponsorship from senior executives
Support from senior management in your organisation will be
important for new AI initiatives to succeed. Your company may
have a Board that strongly favours adopting AI; that sees AI
as over-hyped and irrelevant; or has a healthy scepticism
and seeks validation of benefits before assigning extensive
resources. To build support within your company, define
the focus of your first AI initiative and then set realistic goals.
Your system will not, and need not, offer 100% accuracy. If it
can save effort, even if results require human verification, you
can deliver increased efficiency.
You can then present to senior management a plan
that includes:
• a statement of the problem your AI will solve;
• a summary of outputs and benefits for the company;
• details of the nature and volume of data required;
• a viable approach with realistic timescales.
Leaders may be reluctant to invest in technology they do not
understand. To achieve buy-in, it may be necessary to educate
senior management regarding the benefits of AI while setting
realistic expectations regarding timescales and results.
Anticipate and mitigate cultural concerns
When deploying AI, anticipate the potential for cultural
resistance. For many in your team, AI will be unfamiliar. Some
employees will see their workflows change. Many employees
are concerned about the impact of AI on their job security.
Frequently, AI will enhance an individual’s role by delivering
what is termed ‘Augmented Intelligence’. AI can bring new
capabilities to an employee’s workflow or free a human
operator from repetitive, lower value tasks so he or she can
focus on higher value aspects of their role.
Address concerns proactively by highlighting the ways in
which AI will support individuals’ goals and workflows – and
enable your team to redirect their time to the most engaging
aspects of their roles. “We go through a change management
program to educate the workforce. We explain that AI takes
care of repetitive tasks so people can focus on bigger things”
(Dmitry Aksenov, DigitalGenius).
Address non-traditional security considerations
AI systems can be attacked in non-traditional ways. If a
classification or grouping system is given an input beyond
the scope of the labels on which it has been trained, it may
assign the closest label it has even if the label bears little
relation to the input. Causes of confusion, more broadly, may
be exploited. Malicious individuals have manipulated system
inputs to obtain a particular result, or to disrupt the normal
practise of AI systems (for example, by spraying obscure
road markings to confuse autonomous vehicles).
Protect against malicious activity via thorough system testing
and exception handling, undertaken from the perspective of
an individual deliberately attempting to undermine or exploit
your system.
A long-term strategy should incorporate
evolution and extension
When your first project is underway, anticipate the longer- term
aspects of your AI strategy. Then “obsess about capabilities to
make your vision come true over five to ten years” (Timo Boldt,
Gousto). Your long term AI strategy should consider:
• Maintenance: To maintain your system’s intelligence,
regularly test results against live data to ensure results
continue to meet or exceed your acceptance criteria.
Set aside budget for future updates and retraining and
monitor for performance degradation. Chapter 5 provides
a blueprint for maintaining AI systems effectively.
• Data: “Remember that AI is a capability, not a product.
It’s always improving” (David Benigson, Signal). Monitor
changes in your data over time. As your business grows
or changes focus, data fields (including time series data,
languages and product characteristics) will evolve and
expand. Retraining your system regularly should be a
component of your long-term AI strategy. To develop
a comprehensive data strategy for AI, see Chapter 3.
• Algorithms: AI techniques are developing rapidly; what
you create today may be less accurate and slower than
systems you develop in 12 months’ time using the same
data. Ensure a member of your team understands advances
being made in AI and can advise on when to apply them
to your use cases.
• Scaling: A plan to leverage your existing AI systems by
extending their deployment to additional business units
and geographies.
• New initiatives: A roadmap of new use cases for AI within
your organisation to deliver increased cost savings, greater
revenue or both.
• Legislation: Developments in AI are being monitored by
legislative authorities (see Chapter 6). Develop a strategy
to comply with new legislation as it emerges.
“Plan for the long term
and then obsess about
capabilities to make
your vision come true
over five to ten years.”
Timo Boldt, Gousto
“Remember that AI is a capability,
not a product. It’s always improving.”
David Benigson, Signal
“We go through a change
management program to
educate the workforce. We
explain that AI takes care of
repetitive tasks so people
can focus on bigger things.”
Dmitry Aksenov, DigitalGenius
21
20
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 1
Strategy
People
Chapter 2
Summary
• In AI, job titles vary and can be difficult to interpret.
We describe characteristics and salaries for six key roles:
Data/Machine Learning Engineer; Data scientist; Machine
Learning Researcher; Head of Data; Head of Research/
AI; and Chief Scientist/Chief Science Officer. For each,
individuals’ capabilities vary across competencies in
research, engineering, production and strategy.
• The composition of your team should depend upon the
problem being solved and your approach to doing so. It is
advisable, however, to avoid hiring solo talent. Begin with
a small team, and ensure you have a robust AI strategy in
place before expanding your AI personnel.
• We suggest team structures, first hires and next steps for
six scenarios: “I want insights into internal data”; “I want
to implement third party AI APIs”; “I want to outsource AI
development”; I want to create bespoke AI models”; “I want
to use a combination of bespoke and third party AI”; and
“I have an idea that’s cutting edge.”
• Recruiters, conferences and universities are primary sources
of talent. Traditional recruitment agents find it difficult to
screen AI candidates, so engage with specialist recruiters.
Conferences and meetups are powerful vehicles for talent
acquisition; be active in the AI community, attend and
speak at conferences, and grow your network to discover
capable candidates. Engage with universities; post on their
job boards, establish partnerships and pay for projects to
engage students who may seek future opportunities with you.
• Diversity delivers economic value and competitive
advantage. Review the culture in your company, AI team
and hiring practices to ensure diversity, representation
and inclusion.
• An effective job description should emphasise projects
(the nature of the engagements on which the successful
candidate will work), skills and impact. Most data scientists
seek work that will ‘make a difference’. To attract talent,
demonstrate how the successful candidate’s work will do so.
• When hiring, prioritise adaptable problem-solvers.
In addition to having role-specific and technical skills,
a strong AI candidate will: understand available tools to
enable rapid research and development; appreciate when
to release an imperfect solution and when to wait; and
communicate and collaborate well.
• Optimise every stage of your recruitment funnel.
We provide best practices for: CV screening; phone
screening; technical testing; face-to-face interviews and
post-interview follow-up.
• AI talent is in short supply. Challenge, culture and company
are key for retention. In addition to an attractive financial
package, consider: offering flexible working hours; offering
challenging problems and minimising drudgery through
automation; creating a culture in which diverse ideas are
shared; avoiding ‘lone workers’; ensuring your AI team
receives recognition for its work; and supporting team
members’ publishing and presentation of work.
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
22
23
Chapter 2
People
If you are building an in-house AI team, whether directly or via
recruiters, it will be important to understand the roles you require
and how to attract, deploy and maintain talent. Below, we provide
a blueprint for structuring, building and retaining your AI team.
Hire for required competencies in engineering,
production, research and strategy
Because AI is an emerging field, job titles vary and can
be difficult to interpret. Further, people may describe
themselves in different ways to market themselves for roles
they want. There are at least six core roles in AI. In each,
individuals’ capabilities vary across competencies of research,
engineering, production and strategy (Fig. 5).
Data Engineer/Machine Learning Engineer:
• understands data and can code AI models that are
derivatives of systems already created
• focuses on engineering (creating code for applications
and solutions to go live), not research
• create models but may lack finer understanding to push
the boundaries of research.
Data Scientist:
• focuses on obtaining insight from data using scripts and
mathematical techniques; manipulates data in a variety of
programming languages for solutions to specific problems;
• typically has an academic background to PhD level
• stays current on contemporary research and be capable
of implementing ideas from academic papers
• may lack wider development and AI skills, including
understanding of the needs of live systems; typically
produces reports, not applications.

Deep Learning Researcher/Machine Learning Researcher:
• focuses on research, not building business applications
• highly academic, typically with post-doctoral academic
experience
• seeks to push the boundaries of technical solutions
• will have limited or no exposure of taking their work to
the level of a live application.

Head of Data:
• understands the nuances of varying data sets and is
sufficiently experienced to lead a team
• technically hands-on; works with her team to produce
reports and applications
• may have data strategy responsibilities (responsibility
for acquiring, managing and deriving value from data).

Head of Research/Head of AI:
• research-focused and with enough experience to to lead
a team
• technically hands-on; may be sufficiently experienced to
support the conversion of team output into live applications
Chief Scientist/Chief Science Officer/VP of AI:
• extensive experience in business as well as AI
• determine AI strategy and production pipelines;
work with the Chief Technology Officer (CTO) to ensure
the company’s AI strategy can be executed
• typically report directly to the CEO
• experienced as a board level strategist.
Source: MMC Ventures
Fig. 5. Roles vary across competencies of research, engineering, production and strategy
Role
Data Engineer
Data Scientist
Researcher
Head of Data
Head of AI
Chief Scientist
Research
Low
Medium
High
Medium
High
High
Engineering
High
Medium
Low
Medium
Medium
High
Production
High
Low
Low
Medium
Medium
Medium
Strategy
Low
Low
Medium
Medium
Medium
High
Structure your team effectively
Optimise your hiring process
Invest in retention
People: The Checklist

Clarify the problem to be solved


Identify a development strategy and associated hiring needs

Understand the six core roles in AI teams


Shape your team to reflect the competencies required


Structure your team to avoid lone workers


Understand the role, seniority and requirements for which you are hiring

Develop a clear job description


Leverage recruiters, conferences, meetups, universities and investors


Embed best practices for screening, testing, interviews and follow-up


Identify adaptable problem-solvers


Recruit from diverse backgrounds

Maintain an inclusive and diverse culture

Automate menial work

Offer intellectually challenging problems


Ensure the AI team receives recognition


Support team members’ publishing efforts
25
24
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 2
People
Structure your team according to the problem
and your approach
Salary range and job expectations will vary according to an
individual’s role (Fig. 6).
The composition of your AI team should depend on the
problem being solved, your team’s approach to doing so, and
the level of integration required with your development team
to support production. Bear in mind, however, the following
principles:
• Do not hire solo AI talent, beyond an initial ‘Head of…’ role.
AI professionals rely on collaboration for ideas and can feel
isolated if they are a sole member of a larger team.
• Begin with a small team to validate the data and your team’s
ideas, regardless of the domain.
• Ensure you have a robust AI strategy in place (Chapter 1)
before you expand your team.
“I want insights into internal data”
• Strategy: Build an in-house data science team. Sensitive
data will not leave your company and you can control the
focus and outputs of your team.
» First hire: A Head of Data reporting to your CTO.
» Next step: Hire two or three data engineers or data
scientists, or a combination, depending on the needs of
the project.
» Success factors: These individuals will need time to
understand your data and how it’s gathered. Ensure they
have access to all the data they need.
“I want to implement third-party AI APIs”
• Strategy: You will need individuals who understand your
data and have the knowledge to implement and test third
party APIs. If you have no budget to hire, an alternative
approach is to find existing developers within your
organisation who understand data well enough to
manage the API integrations.
» First hire: Two machine learning engineers.
» Next step: For smaller projects, your engineers could
report into the CTO or Head of Development. For larger
projects, you may wish to hire a hands-on Head of AI as
a team lead, to support the expanding team.
» Success factors: Review the available APIs and validate
that they will address your use cases. Ensure you plan for
changes to the APIs in future.
“I want to outsource AI development”
• Strategy: You need an individual who understands your
project well to manage the outsourced relationship.
» First hire: A Head of AI, if you don’t already have a
suitable person within your team. A Head of AI can also
enable you to bring the solution in-house, over time, if
you choose to do so.
» Next step: Empower your Head of AI to manage project
costs and timelines. Ensure you receive regular status
reports for clarity on each delivery cycle.
» Success factors: Successful AI requires continuous
feedback and iteration. Develop a good relationship with
your AI provider and agree costs for updates up-front.
“I want to create bespoke AI models”
• Strategy: You are undertaking something unique with
your AI solution and wish to keep your data and system
knowledge in-house. This is the most common scenario
for companies.
» First hire: A Head of AI, or Chief Scientist, reporting
to the CTO.
» Next step: Let the Head of AI, or Chief Scientist,
determine your strategy based on the problems you
wish to address.
» Success factors: Allow budget for at least four further
hires – more for larger projects. These hires will be a
combination of Data Scientists and Machine Learning
Engineers. You may need a Machine Learning
researcher as part of this team – but ensure they are
challenged enough.
“I’m going to use a combination of bespoke
and third party AI”
• Strategy: You seek a fast start and third party APIs deliver
enough for your minimum viable product. However, you
want to develop bespoke AI in parallel, to deliver a unique
value proposition.
» First hire: A Head of AI or Chief Scientist, plus at least
two Data Engineers or Machine Learning Engineers to
undertake the API work.
» Next step: Hire two or more Data Scientists.
» Success factors: Unless your hybrid approach involves
examining research, avoid Deep Learning Researchers.
“I have an idea that’s cutting edge”
• Strategy: Validate that your idea is feasible – as well as the
problems and timelines associated with it.
» First hire: A Head of Research or Chief Scientist; two to
three Deep Learning Researchers; plus potentially a Data
Engineer to support them.
» Next step: If required, expand the team with Data
Scientists to balance the team’s skill-set.
» Success factors: Manage timelines closely and be
prepared to assess when research isn’t progressing
to plan and alternative solutions should be explored.
Maintain focus on the goal for the research and avoid
research for its own stake.
Source: MMC Ventures
Fig. 6. The expectations and costs of AI professionals differ
Job title
Salary range (£ thousands)
Data Engineer
Machine Learning
Engineer
Data Scientist
Machine Learning
Researcher
Head of Data
Head of AI
Chief Scientist
45 - 90
60 - 90
45 - 90
60 - 100+
80 - 120
80 - 120
110 - 180+
• Directed problems
• Full access to data
• Create solutions that are deployed live
• Challenging problems
• Mix of APIs and models
• Create solutions that are deployed live
• Generate insights from data
• Create models
• Find new methods
• Solve complex AI problems
• Create new models
• Own data strategy
• Run the data team
• Manage projects
• Define approaches
• Run the AI team
• Manage projects
• Develop and deliver AI strategy
• Board membership
• Autonomy
Job expectations
Structure your team to avoid
lone workers. AI professionals
rely on collaboration.
27
26
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 2
People
Recruiters, conferences and universities are
primary sources of talent
Unless you are a known company, advertising on your own
website is unlikely to be effective. Alternative sources include:
1. Recruiters: Specialist recruiters exist for AI and data science.
Poll your network to identify them. Unlike traditional
development roles, recruitment agents find it challenging
to screen AI candidates due to the research-intensive, data-
specific problems they tackle. High quality AI recruiters have
extensive networks of candidates, can identify candidates
that fit your needs, and can save you more in the cost of your
time than they charge in fees (typically 10%-45% of annual
salary, plus bonuses).
2. Conferences and Meetups: Conferences and meetups
are a powerful vehicle for talent acquisition. Be active in
the AI community and grow your network. Every major city
has an AI network you can join and there are conferences
throughout Europe almost every week of the year. Many
conferences offer job boards, in addition to which you can
meet individuals at the event and undertake an initial screen.
Even if timing does not align between you and potential
candidates, making connections is valuable and you will
become known as a potential employer.

Consider speaking at events – it’s easy to include a “we’re
hiring” closing slide and you may receive extensive interest.
Some conferences require sponsorship for a speaking slot.
This may be appropriate if you are discussing your general
solution in a talk close to a sales pitch. You should never
have to pay, however, if you have an advancement that will
benefit the community. Submit your paper to an academic
conference or one of the many high quality events that
avoid sponsored talks and seek excellent speakers and
and topics.

Academic conferences – including NeurIPS, ICML, ICLR
and AAAI – are busy events at which larger employers are
highly active. These conferences are expensive and smaller
companies can be overlooked. They are, however, valuable
events if you seek exceptional researchers.

Other conferences, including RE•WORK and M3, focus on
the intersection of academic and business applications.
These can be excellent environments in which to meet
individuals who wish to move from academia to industry,
as well as being stimulating environments for general
conversation and ideation.

Local meetups are even less formal and many offer events
specifically for hiring. These are an excellent option if
your team has not spoken before or if you wish to receive
feedback on your approach. If there is not a meetup near
you, start one.
3. Universities: If you are geographically close to a university
with a strong AI department, or have alumni connections
to one, the university may allow you to post on its job
boards. While roles will be focused on students soon to
graduate, alumni can also see the university’s digital job
boards and your role may be passed around networks
of AI practitioners.

You may also be able to work in partnership with a university
by paying for projects. Typically, the head of a laboratory
will accept a project for a fixed cost and the project will
serve as a task for graduate students. Exercise care with this
approach; the university may be motivated by gaining IP
and publications, so ensure your agreement is appropriate.
Students who have worked with you on a part-time basis in
this manner are more likely to seek future opportunities with
you. Similarly, there is a growing trend for Masters and PhD
students to work part-time alongside their studies. If you can
offer flexibility in this regard, you may attract exceptional
candidates who are not seeking full-time work.

Many universities offer excellent AI programmes and
candidates. Examine their research pages and identify labs
working on problems similar to yours. A small sub-set of
universities with high quality AI programs includes:
• UK: Bristol, Cambridge, Edinburgh, Imperial,
Manchester, Oxford, Sheffield, Sussex, UCL.
• USA: Carnegie Mellon, Harvard, MIT, Stanford, Yale.
• Canada: Montreal, Toronto.
• Worldwide: EPFL (Switzerland), Nanyang (Singapore),
Politecnico de Milano (Italy), Technical University of
Munich (Germany), Chinese University of Hong Kong.
4. Investors: If you have secured investment, leverage
your investors. Ask them to introduce you to other AI-
led companies in their portfolio so you can share ideas
and recruitment opportunities. There may be excellent
candidates who are no longer a fit for other companies –
for example, due to a relocation – who would be ideal
for yours.

5. Job Boards: Job boards can be effective at attracting
applications. However, with a public posting on a generic
job board you are likely to receive numerous applications
that are poorer in quality or fit. The time, and therefore
cost, required for an individual in your company to review
them can be considerable. Specialist AI job boards,
including those on Kaggle (www.kaggle.com/jobs) and
StackOverflow (https://stackoverflow.com/jobs?q=AI)
are typically searched only by people already part of these
communities and typically offer higher-quality candidates.
Diversity delivers economic value and
competitive advantage
Many problematic and embarrassing AI systems have been
developed because the teams that created them lacked
diversity. Without different perspectives on data and results,
you may create systems that offer narrow appeal and broad
offence. Review the culture in your company, AI team
and hiring practices to ensure diversity, representation
and inclusion.
Further, in a competitive market for AI talent, leadership in
diversity will enable you to attract exceptional candidates
who may otherwise have overlooked your position – or been
minded to accept an alternative.
Job descriptions should emphasise projects,
skills and impact
When hiring, ensure you understand the role, seniority and the
minimum requirements for which you are hiring. If your team
uses Python exclusively, for example, do not hire someone
who only works in R or Matlab. Missing skills will impact your
costs directly, as individuals take time to address gaps.
Describe the projects on which the successful candidate will
work. Do they relate to computer vision, language, prediction,
generation or other? Use the industry-standard terms of
classification, regression, generative AI, sequencing and
clustering for easier comprehension. Describe the expectation
for the role as well as the difficulty of the problem.
You will also need to sell your company and the domain.
Most data scientists seek work that will ‘make a difference’.
To attract talent, frame your problem to demonstrate how
the successful candidate’s work will do so.
When hiring, prioritise adaptable
problem-solvers
Hiring exceptional talent is challenging. While it can be
tempting to prioritise mathematics candidates with first class
degrees, people with the largest number of academic papers,
or those with the most appointments, consider your company’s
needs. Exclude individuals with poor communication or
collaboration skills, and people who cannot adapt to the fast
pace and fluid nature of industry. In addition to role-specific
skills, a strong AI candidate will:
• have sufficient technical skills to solve AI problems
• understand available tools to enable rapid research and
development processes
• appreciate when to release a solution, even if it is imperfect,
and when to hold back a release
• communicate and collaborate well.
Seek adaptable, intelligent problem solvers – not individuals
limited to following TensorFlow tutorials. Are you recruiting
an individual to undertake research, or do you need
• Title: Deep Learning Researcher
• Role: Working on exclusive medical imagery, you will
be solving classification and generative problems
beyond the current state of the art.
• Team structure: As part of a dedicated AI team
reporting to the Chief Scientist, you will work closely
with Machine Learning Engineers who manage the AI
production pipeline.
• Skills: Python 3.6, including numerical libraries;
Tensorflow; Keras.
Source: MMC Ventures
Fig. 7. Example job description
29
28
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 2
People
someone to obtain insights from data? Depending upon the
problems to be solved, limited academic experience may
be unproblematic if the individual possesses required skills.
Individuals may have gained experience through alternative
initiatives, including hackathons and competitions.
While the interview process for AI roles is similar to the process
for other technical roles, there are differences. A conventional
developer, for example, would undertake a technical test at
a face-to-face meeting. AI candidates cannot demonstrate
their ability to build an AI model at an interview given the
time constraints. Involve existing members of your AI team
throughout the process. The best candidates will complement
existing ideas while bringing something new to your team.
Optimise each stage of the recruitment funnel
1. CV screen
• Maintain your specification for minimum skill-set. While
no candidate will be perfect, identify your red lines and
do not waste candidates’ time, or yours, taking the wrong
individuals forward.
• Prioritise candidates who have stayed over a year in
past roles; it can take months for a new team member to
understand the data specific to your business.
• Evaluate candidates’ ability to contribute to your business
above their academic experience, even for research posts.
An individual with a decade of experience researching
an obscure problem may not adapt to your natural
language challenge.
2. Phone screen
• Identify candidates’ passions and motivations; evaluate if
they will be a good fit for the projects you have planned.
• Ask candidates about their contributions to previous
projects and seek individuals who can explain their
contributions clearly.
• Let the candidates ask you questions for half the available
time; good candidates will want to know as much about
the company, team and projects as you will about them.
• Progress only the candidates who can demonstrate
their passions and exhibit good collaboration and
communication skills.

3. Technical test
• A technical test (Fig. 8) will be expected but it is important
to be reasonable. Do not set a task that requires more than
four hours to complete; candidates’ time is limited and you
should not take advantage of them.
• Offer a problem representative of the work they would
undertake in your team, with (subject to privacy constraints)
real data. Ideally the problem should have a trivial solution,
which will highlight individuals who do not consider data
complexity.
• Technical tests should be specific to the candidate;
individuals can upload information about your tests to
websites, such as Glassdoor, which can give candidates
an unfair advantage. Similarly, recruitment agents may
prime their favoured candidates with the problem upfront.
Kaggle.com can be an appropriate environment in which to
run technical tests.
• For research-based roles, implementing code from an
academic paper may be a suitable test.
• If a candidate offers a solution which demonstrates that the
candidate thinks beyond the trivial, invite them for a face-to-
face interview.
4. Face-to-face interview
• By this stage you should have a small number of exciting
candidates.
• Discuss each candidate’s technical test. Can they critique
their own solutions? What would they do given more time?
These questions will provide insight into how candidates
think and plan their time.
• Add a thought experiment with extension challenges:
how would the candidate solve a problem in sub-optimal
circumstances? What if there are large gaps in available
data, or if data quality varies? What if the business
required a 50% improvement in predictive speed?
Thought experiments will enable you to understand
a candidate’s creativity and how they will perform in a
dynamic environment. If a candidate can only follow steps
described in AI tutorials, their impact on your business will
be limited. Similarly, exercise caution with candidates who
express annoyance when faced with changing business
requirements, or who advocate long timelines for any
change. They may not have the skills and temperament to
thrive in an early stage company.
• For research-led roles, or where you seek a candidate
who will digest state-of-the-art academic papers, ask the
candidate to bring and present a recent paper written
by someone else. Evaluate whether the candidate can
explain another person’s concepts in simple terms. Invite
an interested non-technical person to join this part of the
interview and ask simple questions. Screen out candidates
who cannot communicate the work to the wider business,
or become frustrated with simple questions.
• Discuss, in as much detail as possible, upcoming projects.
See how excited the candidate becomes and prioritise
candidates that are eager and propose solutions.
5. Post-interview
• Provide immediate feedback and, if you still have other
candidates to consider, manage expectations regarding
a decision.
• AI practitioners are data-led and do not appreciate
uncertainty. Come to a decision promptly and make an
offer quickly.
Challenge, culture and company are key for
retention
AI talent is in short supply. When you have attracted high quality
professionals to your team, it is important to retain them.
While an attractive financial package and benefits are
necessary, large companies can – and will – offer higher salaries.
Retain AI talent by catering to team members’ other needs:
• Offer flexible working hours. AI models can take a long
time to run; if they finish at night or during weekends, good
candidates will want to re-engage and alter parameters.
• Offer challenging problems and minimise drudgery.
Steps that can be automated should be. Ensure your team
has support from other parts of the business to do so.
• Ensure your team has appropriate hardware. Crippling your
team to save £1,000 is a false economy.
• Create a culture in which intellectual debate is encouraged
and diverse ideas are shared. Advances in AI are the result
of multiple scientific fields bringing different perspectives to
the same problem. Individuals with different backgrounds
and education see things differently; combining their ideas
will present novel solutions. An environment in which all
opinions can be voiced and debated will enable your AI
team to solve problems faster and motivate your team.
• Regardless of level, ensure your AI team comprises
more than one person. The ‘lone AI worker’ is a frequent
challenge for early stage companies. The scientific nature
of AI instils a need for collaboration and the testing of ideas.
While it can be exciting for an AI team member to be the
first in your company, and to develop your company’s
prototype, months of solo work on your company’s AI
initiative, even amidst collaboration with your broader team,
can be intellectually isolating and drive attrition.
• Ensure your AI team receives recognition for its work.
If individuals have worked for months to develop an
effective AI model, it will be dispiriting for the team that
adds the front-end to receive sole credit.
• Decide early your approach to intellectual property that
projects will produce. Ensure your team understands
whether there is a patent strategy, if team members may
publish results, and if they can present them at conferences.
Many AI practitioners have academic careers they wish to
sustain. If your company can provide support to their efforts
to publish and present, it will be deemed a benefit. Balance
this, however, with an understanding that developing
research to the standard of an academic paper may require
further work that will not benefit your company.
”Write a script to identify and remove
duplicates in the following data set.”
The candidate is given a set of 50 frames from a video.
Some are identical; some close; some have the same
composition but different subjects; and some are unique.

• A strong candidate will understand that this is a data
preparation problem and will consider the impact of
this data on training or testing a model.
• A trivial solution, indicative of a lack of experience,
would be to identify identical images.
• A better solution would group the images, identify a
dominant image from each group as an output, and
discard the rest.
• An excellent solution would understand that the object
of the images may be important and provide a script in
which the important characteristics could be selected.
Source: MMC Ventures
Fig. 8. Example technical test
31
30
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 2
People
Data
Chapter 3
Summary
• For effective AI, develop a data strategy. A data strategy
spans: data acquisition & processing; quality; context;
storage; provisioning; and management & security.
Define your data strategy at the outset of your AI initiative.
• Accelerate data acquisition by using multiple sources.
Developers draw on several sources including: free
resources (such as dataset aggregators); partnerships with
third parties (companies, universities, data providers and
government departments); and create new, proprietary data.
• A high-quality data set has appropriate characteristics to
address your business challenge, minimises bias and offers
training data labelled with a high degree of accuracy.
Develop a balanced data set – if you possess significantly
more samples of one type of output than another, your
system will exhibit bias.
• Primary forms of bias are: unwarranted correlations
(between inputs and output classifications); erroneous
assumptions which cause relationships to be missed
(‘underfitting’); and modelling noise instead of valid outputs
(‘overfitting’). Adjust for overfitting and underfitting by
using different data volumes and model structures.
Remove unwarranted correlations through testing.
• Ensure that the results of your internal testing will be
maintained when applied to real-world data. Test early,
and frequently, on expected live data.
• Managing ‘dirty data’ is data scientists’ most significant
challenge (Kaggle). Smaller volumes of relevant, well-
labelled data will typically enable better model accuracy
than large volumes of poor-quality data. To label data
effectively: consider developing a supporting system to
accelerate data labelling and improve accuracy; draw on
existing AI and data techniques; and seek data labelled by
multiple individuals to mitigate mislabelling.
• Understand the data you use. Ensure you capture the
human knowledge regarding how your data was gathered,
so you can make downstream decisions regarding its use.
Capture data provenance (where your data originated and
how it was collected). Define your variables (differentiate
between raw data, merged data, labels and inferences).
Understand the systems and mappings through which
your data pass to retain detail.
• Store and structure data optimally to support your
objectives. Storage options include basic file-based,
relational, NoSQL or a combination. When selecting
storage plan for growth in data volume, updates,
resilience and recoverability.
• One in three data scientists report that access to data is
a primary inhibitor of productivity (Kaggle). Develop a
provisioning strategy that: ensures data is accessible across
your organisation when needed; contains safeguards to
protect your company against accidents; optimises system
input/output; and maintains data freshness.
• Implement robust data management and security
procedures consistent with local and global regulations.
Personal data is protected by UK and EU law and you must
store it securely. Draw on principles of appropriate storage,
transmission and minimum required access.
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
32
33
Chapter 3
Data
A data strategy will enable your company to acquire, process, govern
and gain value from data effectively. Without a data strategy, your
team’s efforts will be greater than necessary, risks will be magnified
and chances of success will be reduced.
Develop a data strategy for effective AI
“Data is the lifeblood of any AI system. Without it, nothing
happens” (David Benigson, Signal). There are six components
of an effective data strategy (Fig. 9.):
1. Acquisition & Processing: Obtain and process the data
you need to develop effective prototypes and algorithms.
2. Quality: Develop a data set that has the appropriate
characteristics to address your business challenge,
minimises bias and offers training data labelled with a high
degree of accuracy.
3. Context: Understand the provenance of your data and the
mappings through which it passes so you use and share it
effectively within your company.
4. Storage: Store and structure your data appropriately
to support your objectives regarding access, speed,
resilience and compliance.
5. Provisioning: Optimise the accessibility of data to your
team and the implementation of safeguards.
6. Management & Security: Manage data security, access
and permissioning to ensure appropriate use of your
data stores.
Define your data strategy at the outset of your AI initiative.
Review it quarterly and update it as product requirements
change, your company grows or you are impacted by
new legislation.
Fig. 9. The six components of an effective data strategy
Source: MMC Ventures
Fig. X: The six components of an effective data strategy
Ma
nag
ement & Security
Context
Provisioning
Storage
Quality
Acquisition
& Processing
“Data is the lifeblood of
any AI system. Without
it, nothing happens.”
David Benigson, Signal
Formulate a data strategy
Optimise acquisition & processing
Store data optimally
Provision data appropriately
Optimise management and security
Develop a high-quality data set
Understand data context
Data: The Checklist

Develop a data strategy


Review your data strategy quarterly


Ensure your data collection is legal


Preserve detailed fields

Check you have included real world data


Forecast expected growth in data


Evaluate methods of storage and access

Develop and test a resilience plan


Ensure data requests do not block the addition of new data

Develop a plan to archive stale data so access remains fast


Ensure staff have the minimum access they require to perform their role


Use multi-factor authentication

Undertake regular penetration tests to validate your security

Appoint an individual responsible for compliance with legislation

Confirm you have enough examples of data classes for fair predictions

Understand variance in data required to solve your business challenge


Identify sources of bias in your data


Follow best practices for labelling data

Document the sources of your data

Add metadata to capture data collection methods
35
34
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 3
Data
Accelerate data acquisition by using
multiple sources
Obtaining data to develop a prototype or train your models
can be a lengthy process. Ideally, you will possess all the data
you need at the outset and have a data strategy to govern its
access and management. In the real world, neither is likely.
Working on the project may highlight missing data.
“Build access to data at scale from day one” (David Benigson,
Signal). Filling the gaps from your own initiatives can take
months, so use multiple approaches to accelerate progress.
Developers typically draw on several approaches to source
data (Fig. 10) including free resources (such as dataset
aggregators), partnerships with third parties and the creation
of new, proprietary data.
• Use free resources: Evaluate data sources that already
exist and are free to use. Kaggle (www.kaggle.com), a
large community of data scientists and machine learning
engineers, regularly posts data sources for competition
experiments. These can be useful for prototyping and initial
training of machine learning algorithms. Google Dataset
Search (https://toolbox.google.com/datasetsearch) can
help you find specific data sets – be they weather in London
or public transport statistics for Manchester. Further, many
authors of academic papers are now uploading sample
code and data sets (either raw data or locations to acquire it)
to platforms such as GitHub. These data sets are frequently



used for benchmarking. Not all of the datasets from the
above sources are free for business use, so check that your
use of them is appropriate.
• Develop partnerships: Develop partnerships with other
organisations – other companies, universities, data
providers or government departments. Establishing a
mutually beneficial relationship can offer your company
exclusive data and associated benefits.
• Create data: The data you seek may be unavailable or
prohibitively costly. You may need to invest time and
resource to create the data you need – and a quarter of data
scientists do so. The approach – embedding sensors, taking
photos or videos, undertaking surveys or labelling existing
datasets – will vary according to your industry and use case.
Proprietary data is valuable – which is why so little is free.
Developing your repository of proprietary data will yield
value and defensibility over time.
You will need to de-duplicate and merge your data from
multiple sources into a single, consistent store. New data
must follow a comparable process so your data remains clean.
If you merge fields, or decrease the precision of your data,
retain the original data. Being able to analyse gaps in your data
will enable you to plan future data acquisition and prioritise
addressable business use cases.
Develop a balanced, well-labelled data set
A high quality data set has appropriate characteristics to
address your business challenge, minimises bias and offers
training data labelled with a high degree of accuracy.
It is important to develop a balanced data set. If you possess
significantly more samples of one type of output than another,
your AI is likely to exhibit bias. You can decide whether
your system’s bias will tend towards false positives or false
negatives, but bias will be inevitable. There are three primary
forms of bias in AI (Fig. 11):
1. Unwarranted correlations between inputs and output
classification. Systems that offer jobs based on gender
rather than skills, or provide or decline financial products
based on ethnicity, are examples of unwarranted
correlations resulting from unrepresentative input data.
2. Erroneous assumptions in learning algorithms, which
result in relevant relationships being missed – so-called
‘underfitting’ (Fig. 12, overleaf). If underfitting, you have
not sufficiently used the power of your data. If you seek
to predict rental prices for properties, and base your
model only on the number of bedrooms a property has,
your predictions will perform poorly; your model will
ignore important characteristics such as location, whether
a property is furnished, and whether a property offers
parking or a garden.
3. Modelling noise instead of valid outputs – “overfitting”.
An overfitted model takes account of so many details in the
data that it cannot make accurate predictions. Considering
all the health related data of a group of people, for example,
will include so much natural variation in weights, blood
pressures and general levels of fitness that predicting any
characteristics or a new member of the group would
be inaccurate.
Be aware of bias in your data and models to take appropriate
action and minimise its impact. Overfitting and underfitting can
be adjusted with different data volumes and model structures.
Unwanted correlations are frequently more critical to the
business; in addition to erroneous results they can lead to
negative publicity. Test models thoroughly to ensure that
variables that should not affect predictions do not do so.
If possible, exclude these ‘protected variables’ from the
models completely.
Source: Kaggle
Fig. 10. AI developers use multiple approaches to source training data
Other
Government website
GitHub
University/Non-profit
I collect my own data
Google Search
Dataset aggregator
0%
10%
20%
30%
40%
50%
60%
Fig. 10. AI developers use multiple approaches to source training data
6,375 responses
63.7%
33.9%
27.6%
27.6%
23.2%
21.9%
4.0%
Fig. 11. Three types of bias in AI
Source: Victor Lavrenko
Fig. X: Three types of bias in AI
Under-fitting
(too simple to explain
the variance)
Appropriate-fitting
Over-fitting
(forcefitting – too
good to be true)
If you possess significantly more
samples of one type of output
than another, your AI system is
likely to exhibit bias.
“Build access to data at scale from day one.”
David Benigson, Signal
37
36
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 3
Data
If the features you seek are rare, it can be
challenging to achieve a balanced data
set. You wish to develop a model that can
deal with rare occurrences effectively,
but not be overfit. You may be able to use
artificial data, but not when the artefacts
in the artificial data themselves impact
the model. You may also choose to retain
some overfit or underfit bias – and opt
for a greater proportion of false positives
or false negatives. If you err on the side
of false positives, one solution is to let
a human check the result. The bias you
prefer – false positives or false negatives –
is likely to depend on your domain. If your
system is designed to recognise company
logos, missing some classifications may
be less problematic than incorrectly
identifying others. If identifying cancerous
cells in a scan, missing some classifications
may be much more problematic than
erroneously highlighting areas of concern.
Fig. 12. The problem of overfitting
Source: XKCD
It is critical to ensure that the results of your internal testing are
maintained when applied to real-world data. 99% accuracy
on an internal test is of little value if accuracy falls to 20% when
your model is in production. Test early, and frequently, on real-
world data. “If you don’t look at real-world data early then
you’ll never get something that works in production” (Dr. Janet
Bastiman, Chief Science Officer, Storystream). Before you
build your model, put aside a ‘test set’ of data that you can
guarantee has never been included in the training of your AI
system. Most training routines randomly select a percentage of
your data to set aside for testing, but over multiple iterations,
remaining data can become incorporated in your training
set. A test set, that you are sure has never been used, can be
reused for every new candidate release. “When we’re looking
at images of vehicles, I get the whole company involved. We
all go out and take pictures on our phones and save these as
our internal test set – so we can be sure they’ve never been in
any of the sources we’ve used for training” (Dr. Janet Bastiman,
Chief Science Officer, Storystream). Ensure, further, that your
‘test set’ data does not become stale. It should always be
representative of the real-world data you are analysing.
Update it regularly, and every time you see ‘edge cases’ or
examples that your system misclassifies, add them to the test
set to enable improvement.
Data scientists report that managing ‘dirty data’ is the most
significant challenge they face (Kaggle). Smaller volumes of
relevant, well-labelled data will typically enable better model
accuracy than large volumes of poor quality data. Ideally, your
AI team would be gifted data that is exhaustively labelled with
100% accuracy. In reality, data is typically unlabelled, sparsely
labelled or labelled incorrectly. Human-labelled data can still
be poorly labelled. Data labelling is frequently crowdsourced
and undertaken by non-experts. In some contexts, labelling
may also be intrinsically subjective. Further, individuals looking
at large volumes of data may experience the phenomenon of
visual saturation, missing elements that are present or seeing
artefacts that are not. To mitigate these challenges, companies
frequently seek data labelled by multiple individuals where a
consensus or average has been taken.
To label data effectively, consider the problem you are solving.
‘Identify the item of clothing in this image’, ‘identify the item of
clothing in this image and locate its position’ and ‘extract the
item of clothing described in this text’ each require different
labelling tools. Depending upon the expertise of your data
labelling team, you may need a supporting system to
accelerate data labelling and maximise its accuracy. Do you
wish to limit the team’s labelling options or provide a free
choice? Will they locate words, numbers or objects and
should they have a highlighter tool to do so?
Embrace existing AI and data techniques to ease the data
labelling process:
• For visual classification use a generic object recognition
tool, such as ImageNet, to identify relevant categories of
images (such as cars) and the location of the object in an
image. You can then show your labellers the image with a
highlighted area and ask about the highlighted object to
make a deeper classification (such as model).
• For natural language processing you may be able to draw
on existing textual content and classifiers, such as sentiment
analysers, to sort data into broad categories that a person
can verify and build upon for downstream applications.
• Use clustering techniques to group large volumes of
similar data that can be labelled together.
“If you don’t look at real-world
data early then you’ll never
get something that works
in production.”
Dr Janet Bastiman, StoryStream
39
38
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 3
Data
Understand data context by capturing human
knowledge
It is critical to understand the data you use. Using a number
labelled “score” in your database is impractical – and may be
impossible if you do not know how it was derived. Ensure you
capture the human knowledge of how data was gathered,
so you can make sound downstream decisions regarding
data use.
Your data strategy should ensure you:
• Understand data provenance: It is imperative to
understand where your data originated, how it was
collected and the limitations of the collection process.
Does data relate to current customers only, or a spread of
the population? Are the images or audio you use raw or
have they already been digitally edited?
• Define your variables: Defined variables should enable
you to differentiate between raw data, merged data, labels
and inferences (such as assuming an individual’s gender
from their title).
• Understand systems and mappings through which data
have passed. As you process data through multiple systems
and mappings, problems can arise – much as photocopies
of a photocopy begin to degrade. For example, if a system
has a date of birth field which is imported into a system
that requires age instead, the mapping will be accurate at
the time of processing but information has been lost and
the quality of the data will degrade over time. If this is then
mapped to a system that uses an age range, accuracy will
be regained but at the expense of precision. Ensure your
mappings retain detail.
Understanding the context of your data will depend upon
process and documentation more than tooling. Without an
understanding of the context in which data was collected, you
may be missing nuances and introducing unintended bias. If
you are predicting sales of a new soft drink, for example, and
combine existing customer feedback with data from a survey
you commission, you must ensure you understand how the
survey was conducted. Does it reflect the views of a random
sample, people in the soft drinks aisle, or people selecting
similar drinks? It is important to understand the information
not explicitly expressed in the data you use. Documenting
this information will improve your understanding of results
when you test your models. Investigating data context should
prompt your employees to ask questions – and benefit from
their differing perspectives. If you lack diversity in your team,
you may lack perspectives you need to identify shortcomings
in your data collection methodology. Ensure team members
deeply understand your company’s domain as well as its
data. Without deeper knowledge of your domain, it can be
challenging to know what variables to input to your system
and results may be impaired. If predicting sales of computer
games, for example, it may be important to consider
controversy, uniqueness and strength of fan base in addition
to conventional variables.
Store and structure data optimally to support
your objectives
Your data storage strategy will impact the usability and
performance of your data. The nature of your data, its rate
of growth and accessibility requirements should inform
your approach.
Types of storage include basic file-based, relational and
No Structured Query Language (NoSQL):
• Basic file-based: Whether a cloud-based solution – such
as Amazon Web Services (AWS) or HotBlob – or in-house,
basic file-based storage has no limitations on file size – but
is slow to search and search requests are typically based
simply on file name, size or creation date.
• Relational: Relational databases (including MySQL or
Oracle) can store extensive information in separate tables
related to one another. Relational databases are well suited
to defined information, with strict definitions, that can
be grouped into tables. While powerful in their ability to
enable complex queries, and offering security down to
the field level, relational databases can struggle with large
data items (including images and documents) and prove
challenging to scale.
• NoSQL: Recently, NoSQL databases (such as Mongo or
Redis) have become popular because they do not demand
the field restrictions associated with relational databases.
NoSQL databases are effective for storing large volumes
of hierarchical data. Accordingly, they are commonly
associated with ‘big data’ initiatives. NoSQL databases can
easily be scaled by adding extra machines to your system
(‘horizontal scaling’), but struggle to enable complex
queries due to the way in which they store data.
The store you select will influence the performance and
scalability of your system. Consider mixing and matching
to meet your needs – for example, a relational database of
individuals with sensitive information linking to data stored in a
more accessible NoSQL database. The specific configuration
you choose should depend upon the data types you will store
and how you intend to interrogate your data.
To plan for growth and updates:
• Forecast increases in data volume. If starting with existing
data, you will understand current data volumes and how
much new data you are adding each day. If starting from
scratch, you will need to estimate data growth based on a
forecast of incoming data. Armed with an estimate of data
growth, you can determine the volume of storage you will
require for data for the first year of your project.
• Cloud solutions will enable you to store as much data as you
wish – but balance the cost of immediate– and long-term
storage (on AWS, the difference between S3 and Glacier). If
operating your own hardware, you will also need to decide
whether to archive data away from your primary store. You
may need to maintain physically separate data stores for
select personal data, to ensure its isolation.
• Monitor costs, remaining storage, and system performance
so you can act before costs become prohibitive or you run
out of storage space. For relational databases this is critical,
because scaling is likely to require you to upgrade the
hardware on which your database is operating. For NoSQL
systems, it will be easier to scale horizontally.
For resilience and recoverability:
• Treat resilience as mission-critical. Data is the most valuable
component of your AI strategy; if your data were lost, you
could not rebuild your models and would lose a significant
proportion of your company’s uniqueness and value.
• While large companies will have dedicated resources and
specialist skills, startups and scale-ups must also plan for
resilience and recoverability.
• Ensure regular backups. Storage is inexpensive and
accessible to every company.
• The degree of resilience you require will depend upon
whether it is critical for your data store to be permanently
available for read and write. Resilient systems will duplicate
your data, so a replica can take over seamlessly if part of
your system fails. Further, resilient systems typically load
balance to ensure multiple requests do not cause delays.
• Many cloud providers offer resilient systems as part of their
service. While most data centres have their own generators
and redundant internet connectivity, significant events
such as hurricanes and earthquakes can cause hours, or
even days, of disruption. Other risks, including cascading
software failures, can also crystallise. Depending upon the
criticality of your data access you may also seek a separate
provider, with a backup, that you can invoke in the event of
a major disaster. If you manage your own data storage, you
must manage recoverability as a minimum. Store backups in
a separate geographic location and regularly test that you
can restore them successfully. Your first disaster is not the
time to learn that your backups have been failing silently.
Your data storage strategy
will impact the usability and
performance of your data.
Ensure you capture the human knowledge
of how data was gathered, so you can
make sound downstream decisions
regarding data use.
41
40
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 3
Data
When provisioning data consider access,
safeguards and data freshness
One in three data scientists report that access to data is a
primary inhibitor of productivity (Kaggle). Data provisioning –
making data accessible to employees who need it in an orderly
and secure fashion – should be a key component of your data
strategy. While best practices vary according to circumstance,
consider:
• Access: Your data science team will become frustrated if
they are waiting for another team to provide them with data.
Providing them with tools for direct access may be valuable.
Most data stores offer only full administrative access or
expert level tooling. You may need to allow time and
resource to implement a specific solution for your team.
• Safeguards: Protect your company against accidents.
Ensure data access is read-only. Except for an administrator,
no-one should be able to delete or change data.
• Input/output: Reading data from your systems must not
block the addition of new data. Similarly, if your data store
is being continually updated, your team should not have to
wait for a significant period before they can extract the data
they require.
Stale data can be a significant challenge and is a key
consideration when planning your provisioning strategy. If you
are analysing rapidly-changing information, decide how much
historical data is relevant. You might include all data, a specific
volume of data points, or data from a moving window of time.
Select an approach appropriate for the problem you are
solving. Your strategy may evolve as your solution matures.
If you are correlating actions to time, consider carefully the
window for your time series. If you are predicting stock levels,
a few months of data will fail to capture seasonal variation.
Conversely, if attempting to predict whether an individual’s
vital signs are deteriorating, to enable rapid intervention,
an individual’s blood pressure last month is likely to be less
relevant. Understand whether periodic effects can impact your
system and ensure that your models and predictions are based
on several cycles of the typical period you are modelling.
Pragmatically, ensure your access scripts consider the recency
of data you require to minimise ongoing effort.
Implement robust data management and
security procedures
Data management and security are critical components of a
data strategy. Personal data is protected by UK and EU law
and you must store it securely.
You may need to encrypt data at rest, as well as when
transmitting data between systems. It may be beneficial to
separate personal data from your primary data store, so you
can apply a higher level of security to it without impacting your
team’s access to other data. Note, however, that personal data
included in your models, or the inference of protected data
through your systems, will fall under data protection legislation.
Establish effective data management by building upon the
principles of appropriate storage and minimum required access.
• Physical Access: Direct access to your data store should
be tightly limited to key, trusted individuals. Individuals with
the highest level of access to your systems will frequently be
targets for malicious third parties.
• Users: Employees’ needs regarding data access will vary.
If individuals do not need to view sensitive data, they should
not have the ability to view or extract it.
• Applications: Other systems that connect to your data
store should also be treated as virtual users and restricted.
Many companies fail to restrict application access and
suffer adverse consequences when there is an error
in a connected application or the application’s access
credentials are compromised.
Additionally:
• Use multi-factor authentication as broadly as possible.
• Log every access request with the identity of the requester
and the details of the data extracted.
• Hire a third party to undertake penetration testing to
validate the security of your systems.
If an individual resigns, or has their employment terminated,
immediately revoke access to all sensitive systems including
your data. Ensure that employees who leave cannot retain a
copy of your data. Data scientists are more likely to try to retain
data to finish a problem on which they have been working, or
because of their affinity for the data, than for industrial
espionage. Neither is an appropriate reason, however, and
both contravene data protection law. Ensure your team is aware
of the law and that you have appropriate policies in place.
43
42
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 3
Data
Development
Chapter 4
Summary
• There are many ways your company can engage with AI.
Use third party AI APIs; outsource; use a managed service;
build an in-house team; or adopt a ‘hybrid’ approach
combining an in-house team with third party resources.
• Third party AI APIs fulfil specific functions to a moderate
or high standard at low cost. Most solve problems in
the domains of vision and language. Numerous APIs
are available from Amazon, Google, IBM, Microsoft
and also other smaller companies. Features vary; we
provide a summary. APIs offer immediate results without
upfront investment, at the expense of configurability and
differentiation. Use an API if you seek a solution to a generic
problem for which an API is available. APIs are unsuitable
if you seek solutions to narrow, domain-specific problems,
wish to configure your AI, or seek long-term differentiation
through AI.
• Managed services enable you to upload your data,
configure and train models using a simple interface, and
refine the results. Managed services abstract away much of
the difficulty of developing AI and enable you to develop a
custom solution rapidly. Managed services offer greater
flexibility and control than APIs, but less flexibility than an
in-house team, and also require you to transfer data to a
third party and may create dependencies.
• If a third-party solution is unavailable and an in-house team
is too expensive, you can outsource your AI development.
Whether outsourcing is appropriate will depend upon
your domain, expertise, required time to value and data
sensitivity. If outsourcing, specify desired frameworks
and standards, who will provide training data, costs,
timescales and deployment considerations. Outsource if
you require trusted expertise quickly and a cheaper option
than permanent employees. Avoid outsourcing if your
data permissions prohibit it, you require domain or sector
knowledge that an outsourcer lacks, or you wish to build
knowledge within your own company.
• An in-house AI team offers maximum control, capability
and competitive differentiation – at a price. A small in-
house team will cost at least £250,000 to £500,000 per
year. A large team requires a multi-million-pound annual
investment. To develop an in-house team your company
must also: attract, manage and retain AI talent; select
development frameworks and techniques; gather and
cleanse data; learn how to productise AI into real-world
systems; and comply with regulatory and ethical standards.
Build an in-house team if you have a problem that cannot
be solved with existing solutions, seek differentiation in
the market, or seek to maintain control over your data.
• A ‘hybrid’ approach is ideal for many companies. Plan for
an in-house team that will address your requirements to a
high standard over time, but use third party APIs to solve an
initial, simpler version of your challenge. A hybrid approach
can be attractive if you seek rapid initial results, wish to
limit spend until a business case is proven and want greater
differentiation and resilience over time.
• To develop AI yourself you have choices to make regarding
your AI ‘technology stack’. The stack comprises six layers:
hardware; operating systems; programming languages;
libraries; frameworks; and abstractions. Not all problems
require the full stack.
• Ensure your team has hardware with graphical processing
units (GPUs) that support NVIDIA’s CUDA libraries.
Laptops with high performance graphics cards offer
flexibility. For greater power, desktop machines with
powerful GPUs are preferable. To train large models,
use dedicated servers. Cloud-based servers offered by
Amazon, Google or Microsoft are suitable for most early
stage companies.
• Apply AI techniques suited to your problem domain.
For assignment problems consider: Support Vector
Classification; Naïve Bayes; K-Nearest Neighbour
Classification; Convolutional Neural Networks; Support
Vector Regression; or ‘Lasso’ techniques. We describe each
and explain their advantages and limitations. For grouping
problems, explore: Meanshift Clustering; K-Means; and
Gaussian Mixture Models. For generation, consider:
Probabilistic Prediction; Variational Auto-Encoders;
and Generative Adversarial Networks.
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
44
45
Chapter 4
Development
Create a development strategy
Optimise system development
Development: The Checklist
Object detection
✔
✔
✔
✔
✔
✔
✔
✔
✘
✘
✘
✘
✘
✘
✔
✔
✔
✔
✔
✔
✔
✔
✔
✘
✘
✔
✘
✔
✔
✔
✔
✘
✔
✔
✔
✔
✔
✔
✔
✔
✘
✔
✔
✘
✔
✘
✔
✔
✘
✔
✘
✘
✘
✘
✔
✘
Scene detection
Face detection
Facial analysis
Inappropriate content
detection
Celebrity recognition
Text recognition
Written text recognition
Search for similar
images on web
Logo detection
Landmark detection
Food recognition
Dominant colours
detection
Face recognition
(human face identification)
Amazon
Microsoft
Google
IBM
Source: Altexsoft. Check with Amazon, Microsoft, Google and IBM to see their latest features beyond those shown above.
Fig. 13. Image Analysis APIs offer varying features


Review the advantages and limitations of different development strategies


For your AI initiatives, assess the relative importance to your company of time to value, capability, cost, differentiation,
resilience and the development of in-house expertise

Determine the availability of APIs that address your requirements

Assess whether your data permissioning allows use of third party services


Validate that your chosen development strategy offers the trade-offs, integrations with existing systems and resilience
your organisation requires


If developing an in-house team, review best practices with regard to strategy, people, data, development, production
and regulation (Chapters 1 to 6)


Ensure your team has appropriate hardware for rapid iteration and review ongoing hardware requirements

Match the language you use with the rest of your production activity for simplicity and speed

Understand techniques appropriate for your problem domain (generation, assignment, grouping or forecasting)


Experiment with multiple techniques to validate your challenge and highlight characteristics and limitations of
your data


Select a technique that offers the combination of accuracy, development speed and runtime efficiency you require

Maintain awareness of alternative techniques and the pace of their improvement


Select frameworks and libraries to accelerate development based upon your requirements for ease of use,
development speed, size and speed of solution and level of abstraction and control
You may not require a large, in-house team to develop AI. There are many
ways to engage with AI including third party AI APIs, outsourcing, managed
services, creating an in-house AI team, or a ‘hybrid’ approach that combines
an in-house team with third party resources. Extensive AI development,
however, requires knowledge of AI hardware, development frameworks
and techniques. Below, we provide a blueprint for AI development.
We begin by describing the advantages and disadvantages
of different development strategies, so you can identify the
ideal approach for your company.
The purpose and characteristics of AI frameworks (such as
TensorFlow and PyTorch) and popular AI techniques (such as
Support Vector Machines and Naïve Bayes) can be confusing.
To catalyse your experimentation with AI, we then highlight
and explain the AI frameworks and techniques best suited to
solve a range of problems.
APIs offer specific functionality fast
You may be able to solve the problem you have identified by
using an AI application programming interface (API) from a
third party. These services fulfil specific, limited functions to a
moderate or high standard at low cost. API calls can process
your data and provide an immediate result.
Most AI APIs solve problems in the domains of vision and
language. Language APIs include transcription, translation
and topic extraction. Vision APIs include object recognition,
scene detection and logo identification. Numerous AI APIs are
available from Amazon, Google, IBM and Microsoft. Features
vary (Fig. 13-15) and are advancing rapidly.
46
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 4
Development
Object detection
✔
✔
✔
✔
✔
✔
✔
✔
✔
✘
✘
✘
✘
✘
✘
✘
✘
✔
✔
✔
✘
✔
✔
✔
✔
✔
✔
✔
✔
✔
9 languages
✔
✔
✔
✘
✘
✔
✔
✘
✘
✘
✔
✘
✘
✘
✔
✘
✘
✘
✘
✘
✘
✘
✘
Scene detection
Activity detection
Facial and sentiment
analysis
Inappropriate content
detection
Celebrity recognition
Text recognition
Person tracking
on videos
Audio transcription
Speaker indexing
Keyframe extraction
Video translation
Keywords extraction
Brand recognition
Annotation
Dominant colour
detection
Real-time analysis
Facial recognition
Source: Altexsoft. Check with Amazon, Microsoft and Google to see their latest features beyond those shown above.
Fig. 14. Video APIs offer varying features
Amazon
Microsoft
Google
Speech recognition
(speech into text)
✔
✔
✔
✔
100+ languages
✔
✘
✘
✔
✔
✘
✘
✔
✘
✘
✘
✘
✔
6 languages
✔
✔
✔
✔
✔
120 languages
✔
✔
✔
✔
✔
✘
✔
✔
✘
✔
✔
✔
✔
60+ languages
✔
✔
✔
✔
✔
120+ languages
✔
✘
✘
✘
✔
✘
✘
✔
✘
✔
✔
✔
✔
100+ languages
✔
✔
✔
✔
✔
60+ languages
✔
✘
✘
✘
✔
✔
✔
✔
✔
✔
✘
✘
✔
21 languages
✔
Text into speech
conversion
Entities extraction
Language recognition
Topics extraction
Spell check
Autocompletion
Voice verification
Intention analysis
Metadata extraction
Relations analysis
Sentiment analysis
Personality analysis
Syntax analysis
Tagging parts of speech
Filtering inappropriate
content
Low-quality audio
handling
Translation
Chatbot toolset
Key phrase extraction
Amazon
Microsoft
Google
IBM
Source: Altexsoft. Check with Amazon, Microsoft, Google and IBM to see their latest features beyond those shown above.
Fig. 15. Speech and text APIs offer varying features
49
48
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 4
Development
Transferring large volumes of data can become expensive.
If you are using Amazon, Google, IBM or Microsoft for other
aspects of your platform, and your platform provider’s APIs
fulfil your requirements, your existing vendor may be an
attractive option.
Many other companies, however, offer high-quality APIs in the
fields of vision, language and forecasting (Fig. 16). Access a
fuller list of nearly 200 APIs at www.programmableweb.com/
category/artificial-intelligence/api
(source: Programmable Web).
APIs offer immediate, useful results at the expense of niche
functionality and differentiation. APIs deliver:
• Time-to-value: APIs provide immediate capability.
By calling an API, your company can make immediate
use of functions ranging from language translation to
object recognition.
• Low initial cost: While extensive use can become
expensive, APIs can cost as little as tens or hundreds of
pounds to use – making AI accessible to companies of
all sizes and organisations that seek proof of value before
committing to greater budgets.
• Quality: Large companies, including Google and
Microsoft, have invested billions of pounds in their AI
services. Many are highly capable.
• Ease of use: AI APIs are accessible to developers without
expertise in AI. Companies without knowledge of AI can
immediately take advantages of AI via AI APIs.
Limitations of APIs include:
• Functionality: APIs offer specific functionality, often in
the fields of vision and language. If your requirements fall
outside of what is available, an alternative approach will
be required.
• Configurability: APIs do not allow you to adjust the training
data or models on which the services are based. If you wish
to develop services based on unique training data you have,
or tune underlying algorithms for improved results, APIs will
be unsuitable.
• Genericness: APIs are designed for mass adoption; they
tend to be generic and lack depth and domain specificity.
Object recognition APIs can actually tell the difference
between BMWs and Skodas but are unlikely to be able to
tell the difference between a BMW 6 Series and 7 Series.
• Commoditisation: The APIs you use are available to
your competitors. It will be challenging to create lasting
competitive advantage, and associated market value,
through use of third party APIs.
• Lifetime cost: Extensive use of APIs can attract a high
cost relative to an in-house solution you own.
• Dependence: Large vendors have, on occasion,
discontinued APIs. Smaller vendors can be acquired
or cease to operate. Using third party APIs creates a
dependency over which you have no control.
• Privacy: Using APIs involves passing your data to third
parties. Does this comply with your data permissions?
Does the third party retain a copy of your data or use it
for any other purpose?
Overall, APIs are ideal if you seek an immediate, low cost
solution to a generic problem. APIs will be insufficient,
however, if you have a niche challenge, seek greater control
and configurability, or seek long-term differentiation through
AI (Fig. 17).
Many companies adopt a ‘hybrid’ approach (page 54),
using APIs for rapid proofs-of-concept while transitioning to
an in-house team that can deliver improved, differentiated,
domain-specific capabilities over time.
VISION
Clarifai
https://clarifai.com/developer/guide/
• Seek a solution to a generic problem for which a relevant
API is available
• Have limited budget
• Require immediate initial results
• Have limited in-house AI knowledge and resources.
• Seek a solution to a domain-specific or niche problem for
which an API is unavailable
• Have unique training data, or wish to control and configure
your AI, for improved results
• Seek long-term differentiation through AI
• Do not wish to rely on third parties
• Have data permissions that prevent you passing data to
third parties.
EveryPixel
https://api.everypixel.com/
Infinite Loop
http://imagerecognition.apixml.net/
Prisma Labs
https://prismalabs.ai/api-sdk.html
Reconess
https://reconess.com/
Aylien
https://aylien.com/
Indata Labs
https://indatalabs.com/
Meaning Cloud
https://www.meaningcloud.com/
Spot Intelligence
https://www.spotintelligence.com/
Tisane
https://tisane.ai/
Automated Insights
https://automatedinsights.com/
Ayasdi
https://www.ayasdi.com/platform/
Infosys Nia
https://www.edgeverve.com/artificial-intelligence/nia/
Nexosis
https://docs.nexosis.com/
Unplugg
https://unplu.gg/test_api.html
LANGUAGE
FORECASTING
Source: MMC Ventures
Fig. 16. Many additional companies provide AI APIs
Company
Category
Use APIs if you:
Avoid APIs if you:
Website
Source: MMC Ventures
Fig. 17. APIs offer immediate results at the expense
of differentiation
APIs can cost as little as tens or hundreds
of pounds to use – making AI accessible to
companies of all sizes and organisations
that seek proof of value before committing
to greater budgets.
51
50
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 4
Development
• Operation: if an outsourcer builds your models, are you
entitled to deploy them on your own infrastructure – or are
you tied to your outsourcer on an ongoing basis?
Overall if maximising speed and minimising initial costs are
your highest priorities, and APIs are unavailable, consider
outsourcing (Fig. 19).
If outsourcing, specify:
• Frameworks: is there a specific AI framework you require
the outsourcer to use?
• Standards: what accuracy (precision and recall – see
Chapter 5) must models meet?
• Data: will you provide cleaned, labelled training data? Or is
data to be created by the outsourcer?
• Costs: what costs have been agreed?
• Timescales: what timescales must be met? This can be
more challenging than for traditional software development
because improving a model may require experimentation.
• Deployment: how production-ready must the output be?
An in-house team offers differentiation –
at a price
Investing in an in-house AI team offers maximum control,
capability and competitive differentiation – at a price.
An AI team of your own can deliver:
• Flexibility: Control over the hardware, programming
languages, frameworks, techniques and data you employ
offers the flexibility to iterate and expand your solutions as
your needs evolve.
• Capability: APIs offer defined functionality. Managed
service environments limit your ability to tune underlying
algorithms. Outsourced talent will lack your team’s domain
expertise. With an in-house team you have the opportunity
to create optimised solutions, potentially beyond the
current state of the art.
• Differentiation: An in-house team can develop a unique
AI offering that delivers competitive advantage, credibility
in the market and associated value for your company.
• Resilience: Without reliance on third party APIs or
outsourcers, your AI initiatives can enjoy greater resilience
and longevity.
• Security: Retain control over your own data; none of it
needs to be passed to third parties.
Drawbacks of an in-house team include:
• Cost: A small in-house team, comprising two to four people
and the hardare they require, will cost at least £250,000
to £500,000 per year – potentially more to productise the
resulting system. A large team, recruited to solve problems
at the edge of research, will require a multi-million pound
annual investment in personnel and hardware.
• Complexity: To develop an in-house AI team you must
attract, structure, manage and retain AI talent; select the
development languages, frameworks and techniques you
will employ; undertake data gathering and cleansing; learn
how to productise AI into real-world systems; and ensure
compliance with regulatory and ethical standards.
• Speed: It will require months to build a productive in-house
AI team, and potentially longer to collect the data you
require and develop customised solutions that deliver
results to the standard you require.
Managed services offer increased capability
at low cost
Several vendors offer managed AI services. A step beyond
pre-tuned, function-specific APIs, managed services enable
you to upload your data, configure and train your own AI
models using a simple interface, and refine the results.
These services abstract away much of the difficult of
developing AI and enable you to develop a custom solution
rapidly, via a simplified interface and limited coding.
Peak, a leading managed AI service company in which we
have invested, offers an effective solution. Solutions are also
available from Amazon (SageMaker), Google (AutoML), IBM
(Watson), Microsoft (Azure) and Salesforce.
Managed services have several advantages:
• Capability: greater flexibility and control than simple APIs;
managed services enable you to develop custom models
and, potentially, bespoke IP.
• Cost: cheaper than building an in-house AI team or
outsourcing development.
• Speed: faster time-to-value than building an in-house
AI team.
Limitations include:
• Control: less control than in-house development; access to
underlying models will be limited, reducing customisation
and tuning.
• Permissioning: you must be comfortable transferring your
data to a third party.
• Reliance: it may be expensive or unappealing to
migrate away from a managed service provider, given
dependencies and data transfer costs.
• Intellectual Property: Some vendors retain your data to
improve algorithms for all; in other cases, pragmatically
or contractually ownership of the model you develop
may be limited.
If basic APIs will not satisfy your requirements, managed AI
services offer a fast, flexible, way to develop bespoke solutions
at a lower cost than building an in-house team. Managed
services are also ideal for prototyping. If you require more
control, flexibility, autonomy and ownership in time, however,
significant re-development may be required.
Outsourcing offers expertise for moderate
initial investment
If a suitable API or third party product is unavailable, you will
need to build an AI solution. However, investing in an in-house
team is expensive – at least £500,000 per year, typically, even
for a small team. There are cost-effective alternatives. Several
companies provide outsourced AI capabilities, ranging from
contractor developers, who work with your own engineers,
to complete, outsourced AI development.
The nature of AI development enables reseachers to work
on multiple workstreams simultaneously, so outsourcing can
be cheaper than maintaining a permanent team. Conversely,
transferring large volumes of data securely and frequently,
and retraining models on an ongoing basis, can become
expensive. Whether outsourcing is appropriate will depend
upon a range of considerations including:
• Domain: will a third party offer the expertise you require
in your problem domain and sector?
• Expertise: to what extent do you wish to build expertise
in-house?
• Speed: do you require trusted expertise more rapidly than
you can develop it in-house? Do you require a solution more
quickly than you could build in-house?
• Data sensitivity: do you have permission to pass data to
third parties?
• Your challenge is a solved problem but your data is key
• You wish to begin quickly
• Cost is a challenge.
• Require trusted expertise quickly
• Have clarity regarding the solution you require
• Require a cheaper alternative to permanent employees.
• Your data permissions prohibit this approach
• You require extensive control and flexibility
• Speed of response is critical
• Your problem has unique demands.
• Have data permissions that prohibit outsourcing
• Require knowledge regarding your problem domain or
sector that an outsourcer cannot offer
• Wish to build knowledge wihin your company.
Use managed services if:
Use outsourcing if you:
Avoid managed services if:
Avoid outsourcing if you:
Source: MMC Ventures
Source: MMC Ventures
Fig. 18. Managed services offer speed at the expense
of control
If maximising speed and minimising initial
costs are your highest priorities, and APIs
are unavailable, consider outsourcing.
Fig. 19. Outsourcing offers speed at the expense of in-
house knowledge
53
52
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 4
Development
An in-house team may be necessary if your challenge cannot
be solved with existing AI techniques and solutions, if you
face significant restrictions on your ability to pass data to third
parties, or if you seek competitive differentiation through
AI. Before incurring the cost and complexity of an AI team,
however, explore whether alternative methods can deliver
your requirements faster and for a lower cost (Fig. 20).
A hybrid strategy, described below, may be ideal.
To develop an in-house AI team, review all chapters of this
Playbook for best practices regarding strategy, talent, data,
development, production and regulation & ethics.
A hybrid approach can offer the ‘best of both
worlds’
For many companies, a ‘hybrid’ approach to AI is ideal. Plan for
an in-house team that will address your requirements to a high
standard over time, but use third party APIs (or even a non-AI
solution) to solve an initial, simpler version of your challenge.
A hybrid approach may enable you to prove the viability or value
of your idea and justify in-house spend. It can also serve as a
cost-effective way to identify the aspects of your challenge can
be readily addressed and those that will require bespoke work.
“A hybrid approach gives me flexibility. I don’t need to reinvent
the wheel and can focus on doing very specific tasks better than
anyone else in the world” (Dr Janet Bastiman, StoryStream).
A hybrid strategy offers a rapid, low cost start that suits many
companies (Fig. 21). Initial investment in hardware, team and
software can be minimal. Many APIs offer free trial periods
in which you can assess scope for results. Even if your data
restrictions prohibit use of third party APIs, you can adopt a
hybrid approach with in-house developers using pre-trained
AIs. Further, many academic papers and coding competition
entries have code uploaded to GitHub and many have
unrestricted licenses.
If you adopt a hybrid approach, develop a data strategy
(Chapter 1) and pipeline of training data upfront. You can
continue to use third-party APIs if they fulfil your needs unless
costs become prohibitive, you wish to build resilience, or
you seek improved results and differentiation with your own
technology. As you gather additional data, you can create
more accurate and complex models in-house, as needed and
when the business case has been proven.
While the risk of interruption to services from Amazon, Google,
IBM and Microsoft is low, vendors do occasionally remove
APIs. Smaller vendors offering APIs may be acquired, or their
services changed or discontinued. If you adopt a hybrid
approach, develop a strategy for resilience. Once elements of
your product are in place, examine the pre-trained models and
consider moving these services in-house if you can achieve
results comparable with the API. You may be able to use your
chosen APIs in perpetuity and continually develop niche AI to
complement these – a popular approach.
“A hybrid approach gives me
flexibility. I don’t need to
reinvent the wheel and can
focus on doing very specific
tasks better than anyone else
in the world”.
Dr Janet Bastiman, StoryStream
To develop AI, optimise your technology stack
To develop AI – via a managed service provider, outsourcer or
in-house team – you have choices to make regarding your AI
technology stack. The stack comprises six layers: hardware;
operating systems; programming languages; libraries;
frameworks and abstractions (Fig. 22).
We offer hardware recommendations overleaf. The problem
domain you are tackling (assignment, grouping, generation
or forecasting) will then favour particular machine learning
techniques and associated libraries and frameworks. Select
components for your development stack accordingly.
The degree of abstraction you select will depend upon the
skill of your development team, the speed of development you
require and the degree of control you seek over the models
you develop. Greater abstraction offers faster development
and requires less skill, but limits your ability to tune models to
your requirements. The size and speed of your models may
also be limited.
Not all problems require the full stack; some solutions can be
achieved rapidly, without frameworks or abstractions.
For effective R&D, use appropriate hardware
Research and development requires hardware. To process
models quickly, ensure your team has hardware with graphical
processing units (GPUs) that support NVIDIA’s Compute
Unified Device Architecture (CUDA) libraries. These allow
your AI programmes to use the specialised mathematics
of the GPUs and run at least ten times faster than on a CPU.
For many, a laptop with a high performance graphics card is
ideal. Current, potentially suitable cards include the NVIDIA
GTX 1050 Ti, 1070, 1080 and the RTX 2080.
For greater power, desktop machines with more powerful
GPUs are preferable – but at the expense of flexibility for your
team. If you are a multi-premise team, or expect your personnel
to travel, your team may expect a laptop in addition to a
desktop machine you provide for research.
For large models, or rapid training, you will require dedicated
servers. Buying and hosting servers yourself, either on-premise
or in a data centre, is the most cost-effective over the long
term but requires considerable upfront capital expenditure.
The majority of early stage companies will find it more
appropriate to use cloud-based servers offered by large
providers including Google, Amazon and Microsoft. All offer
GPU servers, costed according to usage time. Using the cloud
providers, at least at the early stages of your AI initiatives, will
enable you to control costs more effectively and push the
decision regarding buying hardware to later in the process
when you have a minimum viable product.
• Have a niche problem that cannot be solved with existing
solutions or techniques
• Seek differentiation in the market and associated value
• Wish to retain control over your own data.
• Require rapid initial results
• Wish to limit spend until a business case is proven
• Have an evolving problem and desire for greater
differentiation and resilience over time.
• Have a simple problem for which solutions are available
• Require an initial solution quickly
• Have a modest budget.
• Have a generic problem solved with existing APIs
• Have a complex problem, to which a simple solution will
cause more harm than no solution
• Have data permission challenges that prevent use of APIs.
Use an in-house team if you:
Use a hybrid approach if you:
Avoid an in-house team if you:
Avoid a hybrid approach if you:
Source: MMC Ventures
Source: MMC Ventures
Source: MMC Ventures
Fig. 20. An in-house team offers differentiation – at a price
Fig. 21. A hybrid approach can offer the ‘best of both worlds’ Fig. 22. The six layers of the AI technology stack
Abstractions
Frameworks
Libraries
Languages
Operating System/CUDA
Hardware
(e.g. Keras, Digits)
(e.g. TensorFlow, PyTorch)
(e.g. NumPy, Pandas)
(e.g. Python, R)
(e.g. Linux, Windows)
(e.g. GPUs, CPUs)
Fig. X: The six layers of the AI technology stack
55
54
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 4
Development
Apply AI techniques suited to the problem
domain
For each problem domain (assignment, grouping, generation
and forecasting – see Chapter 1, ‘Strategy’) – there are
numerous established machine learning techniques.
Techniques vary in their data requirements, training dynamics,
deployment characteristics, advantages and limitations. While
deep learning methods are powerful, other techniques may be
sufficient or better suited. Experiment with multiple techniques.
Below, we highlight techniques popular for each domain.
For assignment problems consider SVCs, Bayes,
KNNs and CNNs
Classification problems, which offer a defined, correct output
to ease development, are frequently an attractive starting point
for AI. While convolutional neural networks became popular,
in part, due to their efficacy in solving classification problems
there are many alternative techniques you can apply – many of
which offer effective results and are quicker to implement.
Source: MMC Ventures
Continued on next page.
Technique
Technique
Approach
Approach
Support Vector
Classification
(SVC)
Convolutional
Neural Networks
(CNNs)
Naïve Bayes
K–Nearest
Neighbours
Classification
(KNN)
SVC is effective when classifying images
or text and you have fewer than 10,000
examples. Plot data in multi-dimensional
space, based upon the number of variables
in each example, and the SVC algorithm will
determine the boundaries of each class (Fig.
24). New examples are classified based upon
their relationship to the calculated boundaries.
CNNs comprise multiple layers of neurons.
Data passing through the network is
transformed, by examining overlaps between
neighbouring regions to create areas of
interest. The final layer of the network is then
mapped to target classes.
Naïve Bayes assumes that variables are
independent and is particularly effective
for text classification. Classifications are
developed based upon the probability of
each variable being contained within a specific
class. Probabilities are then combined to
provide an overall prediction.
KNN is a productive statistical technique when
you possess a complete data set. All training
data is mapped into vectors, from an origin
based on the variables in the data. Each point
in space is assigned a label. New data is then
classified by mapping it to the same space
and returning the label of the closest existing
datapoints (Fig. 26).
Effective when there
are many variables.
Memory-efficient.
Excels with complex
data and multiple
output classes.
Fast to train and run.
Effective for text and
variables.
Effective when
boundaries between
classes are poorly
defined.
Prone to overfitting.
Cannot directly provide
probability estimates to
evaluate results.
Compuationally
expensive.
Slow to train.
Highly sensitive to
training data.
Probability for
classifications is
unreliable.
All data must be
stored in memory
for classification;
predictions require
additional resources
and time.
Advantages
Advantages
Challenges
Challenges
Source: Haydar Ali Ismail, (https://bit.ly/2vcSDLf)
Source: Savan Patel (https://bit.ly/2GAYWR5)
Source: Rajeev D. S. Raizada, Yune-Sang Lee (https://doi.org/10.1371/journal.
pone.0069566)
Fig. 24. SVCs maximise the boundaries between classes
Fig. 26. KNNs return the label of the closest datapoint
Fig. 25. Naïve Bayes classifies based on the probability
of a variable being contained in a class
Support
vectors
Support
vectors
Fig. X: Nearest neighbours
Class A
Class B
X1
X2
k = 3
k = 6
Fig. X: Class A Class B
p(x | A)
The probability
of observing x,
if x came from
the Class A
distribution
p(x | B)
The probability
of observing x,
if x came from
the Class B
distribution
(x – µA)/σA
z-score distance of x
from Class A
(x – µB)/σB
z-score distance of x
from Class B
Fig. X: Probability distributions for an individual datapoint
x
While convolutional neural
networks are popular, there are
many alternative techniques you
can apply – many of which are
quicker to implement.
Fig. 23. For assignment problems consider SVCs, Bayes, KNNs and CNNs
57
56
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 4
Development
Regression problems quantify the extent to which a feature
exists. Because they are also assignment problems, the
techniques used for assignment frequently overlap with
those used for regression.
For grouping explore Meanshift Clustering,
K-Means and GMMs
If you have unlabelled data and seek to cluster it into similar
groups, you will require techniques that expose similarity.
Definining similarity can be challenging when there are many
dimensions to the data.
Technique
Technique
Approach
Approach
Support Vector
Regression (SVR)
Meanshift
Clustering
Least Absolute
Shrinkage
and Selection
Operator (Lasso)
K-Means (Lloyd’s
algorithm)
Convolutional
Neural Networks
(CNNs)
Gaussian Mixture
Models (GMMs)
SVR is similar to SVC; training data plotted
in multi-dimensional space. However, unlike
SVC (where hyperplanes are generated
to maximise distance from the data), with
SVR hyperplanes are matched as closely as
possible to the data.
Meanshift clustering discovers groups within
a data set by selecting candidates for the
centre of a group from the arithmetic mean
of the datapoints in the region. The process
continues until there are a distinct set of
groups, each with a centre marker (Fig. 29).
Lasso minimises the number of variables used
to make a prediction. If there are multiple,
correlated variables Lasso will select one at
random.
K-Means groups data into a pre-defined
number of clusters of equal variance (data
spread within the group).
CNNs can also be used for regression
assignment tasks. Unlike when used for
classification, the CNN provides a single
neuron, with the prediction value as an output.
GMMs can offer more flexibility than K-Means.
Instead of assuming that points are clustered
around the mean of each group, GMMs
assume a Gaussian distribution and can offer
ellipse shapes (Fig. 30).
Effective with large
numbers of variables.
Versatile.
Can extrapolate for
new data.
You do not need to
know in advance
how many clusters
you expect.
Fast predictions.
Well suited to situations
in which few variables
are important for a
prediction.
Scalable to large
data sets.
Effective for complex
problems.
Because they draw
upon probabilities,
GMMs can label
datapoints as
belonging to multiple
classes – which may
be valuable for edge
cases.
Prone to overfitting.
The prediction is
provided without
confidence in
its correctness;
confidence must be
determined through
indirect methods.
The algorithm’s
scalability is limited
due to the number of
calculations between
neighbours in each
iteration.
Minimising input
variables may cause
overfitting to training
data.
Selected variables
may oversimplify the
problem.
Defining the number
of clusters in advance
can be difficult because
it requires some
knowledge of the
probable answers.
If data is irregularly
shaped, when plotting
in multi-dimensional
space the algorithm
can become confused
and suggest peculiar
distributions.
Difficult to determine
which inputs contribute
to a prediction.
Difficult to determine
confidence in the
prediction.
If the Gaussian
distribution assumption
is invalid, the clustering
may perform poorly
with real data.
Advantages
Advantages
Challenges
Challenges
Fig. 28. For grouping explore Meanshift Clustering, K-Means and GMMs
Fig. 27. For regression problems, explore SVRs, Lasso and CNNs
Source: MMC Ventures
Source: MMC Ventures
59
58
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 4
Development
Source: Miroslav Radojević (https://bit.ly/2GWfm5W)
Source: John McGonagle, Geoff Pilling, Vincent Tembo (https://bit.ly/2tzlc5k)
Fig. 29. Meanshift Clustering produces distinct groups
with centre markers
Fig. 30. GMMs offer elliptical groupings instead of
assuming points are clustered round a mean
-2
-2
0
0
2
2
4
4
6
6
8
8
10
Fig. X: GMMs offer elliptical groupings instead of assuming points are clustered around a mean
For generation, VAEs and GANs can be
effective
Since its inception, AI has been used to synthesise text; MIT’s
ELIZA natural language processing programme, created
from 1964 to 1966, offered the illusion of understanding
in psychology and other domains. In the decades since,
the quality of generation techniques has been transformed
– particularly following the introduction of Generative
Adversarial Networks (GANs) – while domains of application
have broadened to include visual imagery and sound.
Technique
Technique
Approach
Approach
Pattern matching
Probabilistic
prediction
Variational Auto-
Encoders (VAEs)
Generative
Adversarial
Networks (GANs)
Pattern matching is among the most naïve
of techniques but offers the illusion of
intelligence in text generation. Using a
dictionary of phrases and key words to
recognise input statements, it is possible to
create moderately effective responses with
little effort.
Probabilistic prediction can be effective for
text generation. Given a word or words from
a sentence, probabilistic models determine a
word or phrase to follow and recommend the
text with the highest probability.
VAEs train from real-world data. VAEs use
a convolutional neural network to encode
data into a vector and a second network to
deconvolve the vector back to the original
image (Fig. 32). After training the network,
varying the input vector will provide
realistic outputs.
Generative Adversarial Networks (GANs)
comprise a generator network such as
DCGAN (Deep Convolutional GAN) and a
discriminator network (a standard classification
CNN) (Fig. 33). The generator attempts to
create an output that will fool the discriminator,
while the discriminator becomes increasingly
sophisticated at identifying outputs that are
unreal. With sufficient training, the generator
network learns to create images or text that
are indistinguishable from real examples.
Useful for repetitive
situations that may be
fully mapped – such
as sports reporting
or basic customer
support.
Improve quickly
with use.
Compare the output
directly to the original.
Create realistic
outputs from random
input noise.
Rapidly becomes
nonsensical when
inputs are outside a
predefined area.
Addresses a set of
problems limited
in scope.
The likelihood of a
realistic output decreases
if the difference between
the original data vector
and new input vector
becomes too great.
Image outputs can
be blurry.
Cannot generate
outputs with specific
features unless the GAN
searches the entire input
space. Random inputs
give random (although
realistic) outputs; you
cannot force a specific
output condition.
The discriminator
identifies only real images
and fakes, not whether
the output includes
elements of interest.
The more complex the
image or text being
created, the harder to
create realistic output.
Current research centres
on splitting the challenge
into multiple generative
steps.
Advantages
Advantages
Challenges
Challenges
Continued on next page.
Fig. 31. For generation, VAEs and GANs can be effective
Source: MMC Ventures
61
60
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 4
Development
Source: Kevin Frans (https://bit.ly/2GDnUiJ)
Source: Thalles Silva (https://bit.ly/2MZGKAs)
Source: https://realpython.com/numpy-tensorflow-performance/ (Renato Candido)
Fig. 32. VAEs encode images into a vector and add noise before regenerating
Fig. 33. With one network, GANs generate output from random noise; a second network serves as a discriminator
Fig. 35. Libraries offer improved performance
Fig. 34. For forecasting problems, experiment with causal models, HMMs and ARMA
Fig. X: VAEs – encode images into a vector and add noise through standard deviation before regenerating
Encoder
Network
(conv)
Decoder
Network
(deconv)
Mean
vector
Standard
deviation
vector
Sample
latent
vector
DISCRIMINATOR
GENERATOR
Training set
Random noise
Fake image
Real
Fake
Fig. X: With one network, GANs generate output from random noise;
GANs use a second network as a discriminator
For forecasting, causal models and HMMs
are popular
Applying techniques to predict the future is challenging;
forecasts may be affected by variables outside the data
available to you. While the past is not always indicative of
the future, AI forecasting techniques are effective when there
are causal or periodic effects. Understanding the volume of
data you require may need initial knowledge of causal and
periodic effects, in the absence of which your model may
miss these relations.
Use frameworks to accelerate development
If your team is developing an AI solution, use libraries to
accelerate development. The algorithms described above
have been coded into efficient libraries for Python and R.
Implementing an algorithm directly in Python will be slower –
in some cases 60 times slower (Fig. 35).
Run time for a linear regression problem implemented in pure Python, using TensorFlow (on CPU for
comparability) and using in-built functions in NumPy (a Python library for numerical analysis).
Technique
Approach
Causal models
Hidden Markov
Models (HMMs)
Auto-Regression
Moving Average
(ARMA)
A sub-class of assignment problem, causal
models can use the same techniques – with
the additional consideration of variables’ rate
of change – to predict new values.
Markov models provide a sequence of events
based upon the previous time step. HMMs
assume that predictions of the future can be
based solely upon the present state; further
history is irrelevant.
Despite dating from the 1950s, ARMA remains
useful. ARMA considers past values and uses
regression to model and predict a new value,
while a moving average calculates the error.
A further algorithm determine the best fit for
future predictions.
Straightforward to
implement.
Well suited to learning
and predicting
sequeneces within data
based upon probability
distributions.
Considers past values
and prediction error,
offering greater
adaption than HMMs.
Consider a point in
time; may fail to take
into account longer-
term trends.
Challenging to train.
Rapidly become
inaccurate if
sequences change.
Can oversimplify
problems that have
complex periodicity,
or randomness, in the
time series.
Advantages
Challenges
Implementation
Run time
Pure Python (with list comprehensions)
TensorFlow on CPU
NumPy
18.65 seconds
1.20 seconds
0.32 seconds
While the past is not always indicative
of the future, AI forecasting techniques
are effective when there are causal or
periodic effects.
Source: MMC Ventures
63
62
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 4
Development
For numerical analysis, NumPy is a library
of choice
There are many libraries available to support numerical
analysis. The most popular include:
• NumPy: A library of choice for numerical analysis in Python.
Functions are optimised in C so run quickly, matrices
and vectors are well handled, and there are many in-built
statistical algorithms to support AI.
• Scikit-learn: Released in 2010 as a lightweight library for
deep learning, Scikit-learn is built on NumPy and offers
considerable overlap, although the two complement
each other well.
• Matplotlib: Predominantly a plotting library, to support
visual analysis of plots Matplotlib requires its own numerical
calculations. These are limited and further libraries are
required for broader analytical techniques.
• R packages are not as extensive as those for Python
but there are many for numerical analysis on top of core
R functions – including caret, gimnet, randomForest
and nmle.
In addition to libraries there are specific applications, such
as Matlab and Mathematica, which offer extensive functions.
While popular in academic settings, they are rarely used in
industry given the high cost of software licenses compared
with the free libraries available.
For deep learning, TensorFlow and Caffe are
popular frameworks
Deep learning frameworks are typically more extensive than
numerical analysis libraries and serve as ‘scaffolding’ for your
projects. Your choice of framework will impact speed of
development as well as the features and scalability of
your solution.
With numerous frameworks available, take time to evaluate
your project priorities and the framework best suited to your
goals. The most popular framework may not be optimal
for your initiative. When selecting a framework consider its
advantages and limitations, the skills it requires, availability
of skills, and scaling and speed requirements (both for
development and production).
Unless you are using a pre-trained network, if you
have implemented models in a single framework then
reimplementing them in another will involve retraining from
scratch. You may elect to use multiple frameworks for different
problems, particularly if doing so allows consistency with
existing development languages.
Frameworks evolve at different speeds and, particularly
when maintained by a single business or university, may
be discontinued with limited notice. In a rapidly evolving
field, frameworks with high levels of community support
can be attractive.
Framework
Framework
Features
Features
TensorFlow
Caffe/Caffe2
Theano
MXNet
Torch/PyTorch
DeepLearning4J
Chainer
Digits
Keras
One of the most widely used
frameworks, TensorFlow is
implemented as a Python library,
enabling rapid development of a
wide variety of projects.
There are many example projects
for TensorFlow, and numerous code
samples (available with an open
source license) for different classes
of problem that can be adapted
rapidly for your own tasks.
Caffe is one of the earlier frameworks
implemented in C++ with a Python
interface. Originally designed for
convolutional neural networks, Caffe
grew to support feed-forward networks.
Facebook recently introduced
Caffe2 which is built for mobile,
includes pre-trained models, and
is likely to be merged with PyTorch
(also from Facebook).
Among the oldest deep learning
libraries, Theano is more
mathematical than many and
positioned between analytical
libraries such as NumPy and abstract
frameworks such as TensorFlow.
Much academic research was
undertaken in Theano, with Python,
and many early adopters of AI
employed it.
MXNet supports a wide range of
programming languages including
C++, R, Python and Javascript, and
is maintained by the open source
community.
Torch provides numerous pre-
trained models and development
that is similar to traditional
programming.
While Torch supported only the Lua
language, PyTorch supports Python.
DeepLearning4J is written for Java
and Scala, and supports a wide
variety of networks.
Chainer is a smaller library for
Python, used extensively for natural
language tasks (speech recognition,
sentiment analysis and, translation).
NVIDIA’s framework is freely
available to participants in the
Company’s developer programme
or users of the NVIDIA cloud.
Digits abstracts much of the
programming into a visual interface,
which allows researchers to focus on
network design instead of coding
data import routines or network
architecture components.
Digits will operate with Tensorflow
or on a standalone basis.
Keras is a Python library that
allows rapid prototyping of neural
networks. Not a framework in its
own right, we consider it here
as an extension to Theano and
TensorFlow.
Google
Berkeley
Vision (Caffe)
Facebook
(Caffe2)
University of
Montreal
Apache
Facebook
Eclipse
Foundation
Preferred
Networks
NVIDIA
François
Chollet
High
Medium
Low
Medium
Low
Low
Low
Low
Medium
High
Medium
Medium
Low
Low
Low
Low
Low
Medium
Numerous example
projects are available.
Becoming a standard as
many training courses
use TensorFlow.
Allows lower-level data
manipulation for tuning.
Widely used in the
academic community.
Efficient and scalable.
Straightforward
to implement new
algorithms.
Used for many early
machine learning
courses.
Fast and scalable;
designed for high
performance.
Uses standard
debugging techniques
Supports distributed
training.
Fast and scalable.
Can operate with an
existing Java stack.
Networks can be
modified while running.
Enables rapid
prototyping.
Highly optimised.
Enables rapid
prototyping and
experimentation.
Accessible for beginners
and useful for all levels of
experience.
Significant computational
overhead.
Less efficient than numerical
libraries for certain
calculations.
Challenging to optimise.
Challenging to compile.
Limited support.
No longer developed or
supported.
Developers with experience
may advocate for its use.
Little support in academia or
industry except niche, high
performance use cases.
PyTorch 1.0 was recently
released (October 2018);
change may be rapid.
Limited available Lua talent
for Torch.
Lacking support for Python,
its use is uncommon.
Few examples.
Challenging to debug.
Few examples.
Low levels of support in
academia and industry.
Few available examples.
Restrictive abstraction.
Requires additional
frameworks.
Challenging to customise
networks beyond the
abstraction layer; re-working
may be required to utilise
underlying frameworks.
Community
Support
Community
Support
Maintained
by
Maintained
by
Availability
of Talent
Availability
of Talent
Advantages
Advantages
Challenges
Challenges
Continued on next page.
Fig. 36. Different deep learning frameworks offer advantages and challenges
Source: MMC Ventures
64
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 4
Development
Production
Chapter 5
Summary
• An unused AI system delivers no value. Develop a
production process that smoothly transitions AI systems
you have in development to live use.
• AI production follows a conventional development
process and requires you to undertake research, develop
a prototype and create a minimum viable product (MVP).
Once in production, undertake cycles of ideation, research,
development and quality assurance.
• Effective R&D requires rapid iteration. Initially, optimise
for speed over quality. Releasing an early model into
production for feedback is preferable to waiting until
a research model is perfect.
• During the R&D phase, solicit feedback about prototypes
from beyond the AI and production teams to minimise
expensive, later redevelopment.
• When moving from MVP to production, select an
appropriate hosting environment. On-premise hosting is
suitable for those with highly sensitive data and existing
on-premise hardware, but is rarely preferred by early stage
companies given high upfront costs, unpredictable activity
levels and required security expertise. Hosting your own
hardware in a data centre offers control and value over
the long term. Upfront costs can be high, however, and
managing a data centre can prove a distraction for young
companies. Cloud hosting, which offers low upfront costs
and high levels of flexibility, is well suited to many early
stage companies – although annual costs can be double
that of a self-managed data centre and cloud hosting
may be unsuitable for highly sensitive data. Consider the
physical location in which your cloud servers are hosted.
Different countries have varying rules regarding data
and you may be required to keep your data within its area
of origin.
• Proving that AI systems are effective differs from the typical
software quality assurance (QA) process. Test your AI
system at multiple stages – during training, validation and
continuously through its life. Efficiency is critical; automate
testing to as great an extent as possible.
• Understand the three common measures of ‘accuracy’ in
AI – recall, precision and accuracy – and monitor all three
to capture performance. Balancing precision and recall is
challenging. Whether you elect to minimise false positives
or false negatives should depend upon the nature of your
sector and the problem you are solving.
• An effective maintenance programme will sustain your AI’s
intelligence. Beyond the maintenance you would typically
perform on a software system, you should verify and
update your AI system on an ongoing basis. AI technology
is developing at pace. Invest in continual improvement to
ensure your system avoids obsolescence.
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
66
67
Chapter 5
Production
An unused AI system delivers no value. It’s essential to develop a
production process that smoothly transitions the AI systems you have
in development to live use. Below, we describe an optimal production
pipeline – in which rapid iteration, appropriate hardware, suitable
hosting, rigorous testing and ongoing maintenance deliver high
quality results.
AI production follows a conventional
development process
By the time you are considering how to take your solution live
you should:
• Understand how you are going to solve the business
problem
• Possess the required data
• Know the development languages and frameworks you
will use
• Appreciate whether hardware with Graphical Processing
Units (GPUs) will be required
• Understand whether your live system will be powered by
local hardware or the cloud.
The aforementioned will enable you to determine an optimal
process to move from development to a live solution – the
production environment.
Progressing an AI system from idea to reality should follow
a broadly conventional development practise – although
timescales may be less certain. After ideation, undertake
research, prototype and then develop a minimum viable
product (MVP). Once in production undertake cycles of
ideation, research, development and quality assurance.
Source: MMC Ventures
Fig. 37. The AI production pipeline is similar to a normal development practice
IDEA
QA
RESEARCH
PRODUCTION
MVP
Fig. X: Title TBC
R&D
IDEAS
FEATURES
PROTOTYPE
Optimise Research & Development
Create a rigorous testing process
Establish an effective maintenance programme
Prototype effectively
Develop efficient Production and Deployment workflows

Clarify the required characteristics of your planned system


Limit abstract research – focus resources on solutions to business problems


Identify system demands and associated RAM/GPU requirements


Leverage existing infrastructure, if present


Report against all measures of accuracy (precision, recall, accuracy)


Establish definitions for ‘better’ models


Automate testing to as great an extent possible


Validate live results frequently


Test edge cases


Establish defined downtime and update procedures


Identify relative system priorities (speed, precision, recall)

Create production-ready code, even for prototypes


Solicit feedback from people outside the AI team

Understand the additional development required for an MVP


Establish a controlled release process


Plan for rapid deployment of code and models

Co-deploy models and related code

Anticipate continual cycles of improvement


Select an appropriate hosting environment


Establish increasing automation over time


Ensure problematic deployments can be reversed
Production: The Checklist
69
68
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 5
Production
For effective R&D, iterate rapidly and use
appropriate hardware
Whether you have an in-house team or are outsourcing, ensure
the team understands the characteristics required from the
end system. Which considerations are flexible? Which are not?
Different decisions will be made if speed is more important
than accuracy – and vice versa.
Even in the research phase, ensure that development is
undertaken in the language used for deployment. If you plan
to deploy in Python, for example, avoid the overhead of
rewriting models created in MatLab or R.
Initially, optimise for speed over quality. It’s better to release
an early version of a model from the research environment
into production, and then to solicit feedback in the live
environment, than to wait until the research model is perfect.
“Spend a month to get a weak model and then iterate to make
it great” (Eddie Bell, Director of Machine Learning, Ravelin).
Isolating models within the research environment will push
considerations of deployment, usability, performance and
scalability to the end of the project instead of addressing them
early. In addition, it increases the risk of a model performing
poorly with unexpected real-world data. Many data scientists
resist releasing models that are not ‘good enough’. Overcome
this hurdle by developing a culture in which the dynamics of AI
development are understood and people are not blamed for
early, poor quality results.
Effective research & development requires appropriate
hardware – see page 55 for guidance.
Ensure your AI team has its code in a source control system
– Git, Mercurial and Subversion are popular – and update it
regularly. The size of trained models can exceed file size limits
on these systems. If file size is a constraint, find an alternative
way of versioning and storing your files. A simple solution (for
example, creating zip files on a shared drive) can be effective
but ensure these files are regularly backed up to prevent
accidental deletion or changes breaking your AI models.
Your research team may find that it is creating many similar
models – for comparable problems or multiple clients.
Automate repetitive tasks to as great an extent as possible,
with your Research team validating the results and using their
specialised skills to adjust the network architectures.
Develop prototypes and solicit feedback
beyond the AI team
During the research and development phase, your non-AI
development and production teams should take the AI models
you have in development and insert them into environments in
which the models may be used.
These early prototypes will be incomplete and unfriendly for
users, but will show the capacity for AI to solve the problem.
Before your system can become a minimum viable product
(MVP), prototypes will highlight related development work
required – including website changes, the creation of
database connections, mobile application modifications or
development of application programming interfaces (APIs).
Prototyping will engage stakeholders, allow other applications
to call the model, enable initial scrutiny of results, and serve as
a starting point for improvement.
During the prototype phase it is critical to solicit feedback
from people outside the AI and production teams. Begin with
internal stakeholders and, with each improvement, move
closer to feedback from end users. Are your models:
• adequately performing their intended function?
• as fast as they need to be?
• scaling with usage as required?
Answering these questions early will avoid expensive
redevelopment later. As with developing non-AI systems,
frequent and iterative changes offer flexibility to address
difficulties as they emerge.
Before your team completes the research and development
iterations that feed your prototypes, finalise plans for a release
process and for deploying code to its final environment.
The number of stages in this process, and its complexity, will
depend on factors including: the importance of controlling
the code (processes for code review, testing, code merging,
build, and versioning); the implications of system downtime;
and the level of automation you require.
Considerations are company-specific – but evaluate:
• During development, will your code and models be tested
with every update or only when there is a viable release
candidate?
• Will testing be automated? How frequently will tests be
updated?
• Does a successful test trigger a live deployment? What
manual steps are required for a system to be made live?
• Will you deploy code directly or create containers? How will
large AI model files be deployed? Will system downtime be
required to make new versions live?
If you have existing development practises, adopt these to the
extent possible to ensure that AI is not considered separately
from the rest of your team’s development efforts.
While automating release based on certain metrics may be
straightforward, understanding whether a new AI system is an
improvement overall may be difficult. A new version of your AI
system may offer improved accuracy at the expense of speed,
or vice versa. Whether you are automating deployment or
verifying it manually, prioritise what is important to your
use case.
When moving from MVP to Production, select
an appropriate hosting environment
With an initial model, supporting code and an established
deployment process you should have a minimum viable
product (MVP) ready for release to your production (live)
environment. The MVP is distinct from your prototypes – while
imperfect, it will contain all the elements required to solve the
problem including peripheral components (web interfaces,
APIs, storage and versioning control).
Having a model in production does not mean it needs to be
publicly visible or impact live results. It should, however, be
exposed to live data so your team can make refinements until
it meets the requirements for a major release. With live data
you can undertake longer-running tests and provide your data
science team with feedback on what is working well and what
is not. At this stage, prioritise establishing a controlled release
process with thorough code testing, and the stability of your
solution. You should also monitor the performance
and scalability of your system.
Plan continual cycles of improvement – investigate and
implement ideas for iterating the model, changing the
interface and responding to feedback. New models must
be demonstrably superior to old ones. Test all changes
before updates are released to the production environment,
allocating work between the AI team and the general develop-
ment team. These cycles will continue for the life of the system.
If you’ve yet to decide where your system will run – on premise,
in a data centre or in the cloud – at this point you will have the
information you need to select an environment, and hardware,
that are suitable for your needs.
On-premise: If your data is highly sensitive and cannot leave
your network, or you wish to keep data and inferencing entirely
under your control, you may wish to host your AI systems
within your own premises. Usually, this is possible only for
companies that have their own internal hardware infrastructure
already. This can be a highly cost-effective option if the
volume of requests to manage is known and relatively stable.
However, all new hardware must be ordered and provisioned,
which will limit scalability. Further, security will be entirely
your responsibility. As such, on-premise deployment is a less
preferred option for early stage companies that will lack these
specialised skills.
“Spend a month to get
a weak model and then
iterate to make it great.”
Eddie Bell, Ravelin
71
70
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 5
Production
Source: MMC Ventures
Source: MMC Ventures
Fig. 38. When to use on-premise deployment
Fig. 40. When to use cloud deployment
Use on-premise if you:
Use this approach if you:
Need to fix your costs
Need flexibility in resource
Have existing on-premise hardware
Have existing systems and data in the cloud
Are working on highly sensitive data
Have limited capital to get started
Do not have robust in-house security expertise
Cannot guarantee volumes of requests
Need your models to be accessed from outside your network
Avoid on-premise if you:
Source: MMC Ventures
Fig. 39. When to use a data centre for deployment
Use this approach if you:
Wish to fix your costs
Have existing data centre hardware
Seek control over your data
Require flexibility in your resourcing
Wish to avoid high up-front capital costs
Avoid this approach if you:
Data centre: If you can afford the capital expense of buying
servers, and have limited need to scale rapidly, hosting your
own hardware in a data centre – either your own or a third
party – can be an attractive option. The cost, even over a
year, can be far lower than using a cloud service and you will
maintain control over your system’s performance. Using a
data centre can also be sensible when you already have large
volumes of data on your own servers and wish to avoid the cost
of uploading the data to a cloud service.
The capital expense of establishing and managing a data
centre can, however, be high – although for early stage
companies there are programmes, such as NVIDIA Inception,
which offer discounted hardware. As with the on-premise
option, only consider a data centre approach if your company
already has its own data centre for other servers, as well as staff
with skills to install and configure your hardware. In addition to
the upfront cost, the distraction of managing a data centre may
prove an inappropriate and unwelcome distraction for your
early stage company focused on its core initiatives.
Cloud: For good reason, many early stage companies choose
cloud hosting from the outset. Amazon AWS, Google Cloud,
Microsoft Azure and Rackspace are popular cloud providers.
The majority of cloud providers offer specialised options
so you can begin quickly and need set up little more than a
security layer and links to other systems in their cloud.
Further, time-based costings allow rapid upscaling and
downscaling of resources as required. For companies
without dedicated system administrators, cloud may be an
easy choice. You will, however, pay a premium for the service.
A year of cloud hosting can cost twice as much as hosting
in a data centre. You will also pay to transfer data in and out.
Nonetheless, costs are payable monthly, rather than as a single
large capital expenditure, and you will also avoid the cost of
staff to manage the hardware.
Unless there is a compelling reason to do so – cost, location
or you are replacing a supplier – it is usually desirable to use
the same cloud provider for your AI hosting that you use for
your other infrastructure. This will limit data transfer costs
and provide a single infrastructure to manage for security
and resilience.
Although cloud systems offer extensive physical security,
be aware that you are placing your software and data on
the internet. If you do not secure the cloud servers that you
establish, you will be at risk. Your cloud provider should ensure
that: access to their data centre is secure; their data centre has
multiple power sources and internet connections; and there is
resilience in all supporting infrastructure such that the provider
can resist any direct security challenge either in person or via
attempted hacks into their network. They should also ensure
that the data images they provide to you are secured from
the rest of their infrastructure and other customers’. It is your
responsibility, however, to ensure that your systems on their
infrastructure are secure. Direct access to your account should
only be via multi-factor authentication – not a simple username
and password. Data stored should be private and any external
data access or calls to your AI must be established using
best practices for authentication. There are many malicious
individuals who scan the IP addresses registered to cloud
providers, looking for unsecured systems they can exploit.
Finally, consider the physical location in which your cloud
servers are hosted. Different countries have varying rules
regarding data and hardware. You may need to keep your data
within its area of origin. Be aware of local laws that could allow
the cloud servers to be restricted. US law for example, allows
hardware from a cloud provider to be seized if authorities
suspect its use for criminal activity. If you are unlucky enough
to have data on the same physical system, you could lose
access to your systems without notice. This risk can readily
be mitigated with appropriate monitoring of your remote
systems and images of your servers that you can start in other
zones if required. Finally, different regions may have varying
performance at different times of day – a dynamic you can use
to your advantage.
Already have systems and personnel established in a data centre
Use highly sensitive data
Avoid this approach if you:
For good reason, many early
stage companies choose cloud
hosting from the outset.
73
72
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 5
Production
Test for precision, recall and accuracy at
multiple stages
Proving that new AI releases are effective, and an improvement
on prior versions, differs from the typical software quality
assurance (QA) process. Test your AI system at multiple stages:
• During training: While your model is being trained,
constantly test it against a subset of training data to validate
its accuracy. The results will not represent the performance
of the model fully, because the randomised test data will
have influenced the model. As a result, this testing will
overstate the model’s accuracy.
• During validation: Set aside a part of your training data
for validation. This test set – known as the validation set – is
never used for training. Accordingly, the predictions your
AI system makes from the validation set will better represent
the predictions it makes in the real world. Validation
accuracy is usually lower than training accuracy. If your
data set does not represent real world data well, however,
validation accuracy will still over-report the accuracy of
your model.
• Continuously: Once your model has been created,
test it against live data for a more appropriate measure
of accuracy.
”Accuracy” has a specific meaning in AI – but, confusingly,
is also used as a general term to cover several measures.
There are three commonly-used measures of accuracy in AI:
recall, precision and accuracy. Understand these measures
to decide which are important for your systems so you can
validate them appropriately.
Consider an AI that determines whether an apple is ‘good’
or ‘bad’ based on a picture of the apple. There are four
possible outcomes:
1. True positive: The apple is good – and the AI predicts ‘good’.
2. True negative: The apple is bad – and the AI predicts ‘bad’.
3. False positive: The apple is bad – but the AI predicts ‘good’.
4. False negative: The apple is good – but the AI predicts ‘bad’.
Using the example above:
• Recall: What proportion of the good apples did I
find correctly?
The number of correctly identified good apples divided
by the total number of good apples (whether correctly
identified or not).
• Precision: What proportion of the apples I said are good,
did I get right?
The number of correctly identified good apples divided
by the total number of apples labelled as good (whether
correctly identified or not).
• Accuracy: What proportion of the apples did I label
correctly?
The number of apples correctly identified as good or bad,
divided by the total number of apples.
Avoid the temptation to use a single measure that flatters
results. You will obtain a truer picture by using all three
measures.
Balancing precision and recall can be difficult. As you tune
your system for higher recall – fewer false negatives – you will
increase false positives, and vice versa. Whether you elect to
minimise false negatives or false positives will depend on the
problem you are solving and your domain. If developing a
marketing solution, you may wish to minimise false positives.
To avoid the embarrassment of showing an incorrect logo,
missing some marketing opportunities may be acceptable.
If developing medical diagnostics, on the other hand, you may
wish to minimise false negatives to avoid missing a diagnosis.
Automate testing to as great an extent as possible. Every new
model should be tested automatically. “Efficiency is critical.
If you have to do something more than once, automate it.”
(Dr. Janet Bastiman, Chief Science Officer, Storystream). If all
measures of accuracy are higher, the decision to deploy the
new model will be straightforward. If measures of accuracy
decrease, you may need to verify the new model manually.
A decrease in one measure of accuracy may not be
problematic – you might have re-tuned your model for
precision or recall, or decided to change the entire model
to improve performance. If your models produce results that
are concerning, speak to your AI team to discuss why.
It may be that your training data set does not contain enough
appropriate data. If you encounter problems, add examples
of these types of data to your test set so you can monitor
improvements.
An effective maintenance programme will
sustain your AI’s intelligence
A deployed AI solution reflects a point in time; available
data, business requirements, market feedback and available
techniques will change. Beyond the typical maintenance you
would perform on any software system, you need to verify
and update your AI system on an ongoing basis. Once your
solution is live, ensure it continues to perform well by:
• Continuously sampling its result and verifying that the
outcome from your model is as you expect from live data.
• Adding problematic data to your test set to ensure your
team has addressed issues.
• Exploring whether new, third-party APIs are available which
outperform your model, which will enable you to focus your
AI team’s efforts onto harder problems.
• Ensuring your system issues alerts if incoming data fails, so
problems can be addressed.
AI technology is developing at pace. Further, the varieties and
volume of available training data continue to evolve. Invest
in continual improvement to ensure the system you develop
today avoids obsolescence.
Available data, business
requirements and techniques
will change over time. Invest in
continual improvement to avoid
obsolescence.
75
74
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 5
Production
Regulation & Ethics
Chapter 6
Summary
• As consideration of data privacy grows, and with the
General Data Protection Regulation (GDPR) in force across
the European Union (EU), it is vital to ensure you are using
data appropriately. The GDPR applies to all companies
processing the personal data of people in the EU,
regardless of a company’s location.
• Companies that are ‘controllers’ or ‘processors’ of personal
information are accountable for their handling of individuals’
personal information. Demonstrate compliance with
GDPR data handling requirements and the principles of
protection, fairness and transparency.
• Minimise the personal data you require, to reduce
regulatory risk, and pseudonymise all personal data through
anonymisation, encryption or tokenisation.
• In addition to standardising data handling requirements and
penalties for misuse, the GDPR introduced considerations
that can impact AI systems specifically. Verify that
automated systems meet GDPR stipulations. Article 22
of the GDPR prohibits legal effects that result solely from
automated processing being undertaken without an
individual’s explicit consent, when consent is required.
Several legislative terms are subject to interpretation at this
time. It may be prudent to make your system advisory only,
and include a human check, if you are developing a system
that could materially impact an individual’s life.
• ‘Explainability’ – explaining how the outputs of your AI
system are derived – is growing in importance. Convention
108 of the Council of Europe, adopted into UK and EU law
in May 2018, provides individuals with the right to obtain
knowledge of the reasoning underlying data processing
systems applied to them. Explainability can be challenging
in relation to deep learning systems. Explore varying
approaches to explainability including Inferred Explanation,
Feature Extrapolation and Key Variable Analysis. Each offers
trade-offs regarding difficulty, speed and explanatory power.
• Develop a framework for ethical data use to avoid
reputational and financial costs. The ALGOCARE
framework, developed by the Durham Police Constabulary
in partnership with academics, highlights issues you should
consider when managing data. It incorporates: the nature
of system output (Advisory); whether data is gathered
lawfully (Lawful); whether you understand the meaning of
the data you use (Granularity); who owns the IP associated
with the data (Ownership); whether the outcomes of your
system need to be available for individuals to challenge
(Challenge); how your system is tested (Accuracy);
whether ethical considerations are deliberated and stated
(Responsible); and whether your model has been explained
accessibly to as great an extent as possible (Explainable).
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
76
77
Chapter 6
Regulation & Ethics
As consideration of data privacy grows, and with the new General
Data Protection Regulation (GDPR) in force across the European Union,
it is important to ensure you are using data appropriately. Today, data
permissioning and management are critical aspects of any AI-driven
company. Below, we describe regulatory and ethical considerations to
help you manage safely the data you use to build your models. Seek
legal advice to ensure compliance with any applicable legislation; the
information below is introductory in nature and will not reflect your
company’s individual circumstances.
Ensure compliance with GDPR data
handling requirements
The GDPR came into force across the European Union on 25th
May 2018. It applies to all companies processing the personal
data of people in the EU, regardless of a company’s location.
Among other considerations, it standardises data handling
requirements and penalties for data misuse. Article 83 of
the GDPR specifies fines of up to 4% of a company’s global
revenue or €20m – whichever is greater – for non-compliance.
Individuals, organisations and companies which, according to
the GDPR, are either “controllers” or “processors” of personal
information are accountable for their handling of individuals’
personal information. Companies must “implement measures
which meet the principles of data protection by design and
by default”. Transparency and fairness are also key concepts
within the GDPR. You must be clear and honest regarding
how you will use individuals’ personal data – and must not use
personal data in a way that is unduly detrimental, unexpected
or misleading for the individuals concerned.
Demonstrate compliance with the GDPR principles of
protection, fairness and transparency in multiple ways,
including by:
• Collecting only the data you require
• Being transparent regarding why data is collected, what
it will be used for and who will have access to it
• Ensuring you have appropriate permissions to store and
process your data
• Removing unnecessary personal data
• Deleting data when its agreed purpose has been fulfilled
• Anonymising data, where possible, to remove personal
identifiers
• Encrypting personal data
• Securing physical access to your data storage
• Limiting access to your data
• Monitoring data access and saving an audit trail of
individuals who have viewed or changed data
• Using data only for the purposes agreed
• Implementing a process to provide an individual with a
copy of all the data you hold about him or her
• Implementing a process to remove all the data you hold
about a specific individual.
The GDPR has expanded the definition of personal data,
which broadly refers to information relating to an identified
or identifiable person, to include information “specific to the
physical, physiological, genetic, mental, economic, cultural
or social identity of that person”. This includes job titles,
employers and social media handles – even if the individual
has made these public on selected websites. While some
information may directly identify an individual, other
information may do so indirectly if combined with other
information. In both circumstances, the data is deemed
personal information. Further, if you are inferring personal
information – such as gender, age or salary – using your
system, you must treat the information as if it were gathered
from the user directly.
Demonstrate compliance with GDPR
principles of protection, fairness and
transparency in multiple ways.
Comply with regulations and license requirements
Deliver explainable, ethical AI


Review your compliance with current legislation including the UK Data Protection Act, the EU GDPR
and EU Convention 108

Monitor proposed legislation to anticipate implications


Review permissions for the customer data you collect

Check that the data sets you use are available for commercial use


Validate licenses for open source models you use

Understand industry-specific explainability requirements

Define an explainability framework


Select and apply an approach to explainability

Update your framework documentation as your models and data change


Validate that your use of data is ethical
Regulation & Ethics:
The Checklist
79
78
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 6
Regulation & Ethics
Certain personal data – in categories including racial origin,
religious beliefs, genetic data and data concerning a person’s
sexual orientation – may be considered “sensitive data” and
require special care. Pseudonymise all personal data through
anonymisation, encryption or tokenisation:
• Anonymisation: Remove or replace personal data with
random information. Even if unauthorised individuals read
the data, they will be unable to identify the data subject.
• Encryption: Encrypt personal data fields. The decryption
key will be required to identify an individual from the
encrypted data. The decryption key must be stored
safely. AI techniques remain effective on encrypted data,
enabling you to identify patterns even if the input data is not
human-readable. This offers a way to incorporate personal
attributes more safely.
• Tokenisation: Remove personal data from the main data
set and replace it with numerical tokens that relate to each
aspect of personal data. The process may be as simple
as providing each individual with a unique identifier. The
personal data and corresponding token are stored on a
separate, more secure, system, allowing the data to be
reconnected at a later date. Tokenisation is effective when
one party has permission to view personal data and needs
to interpret the results of the AI system, but the company
providing the AI system does not need to view the personal
data – for example, a medical analysis system.
Even with security best practices in place, holding personal
data remains a risk. Minimise the personal data you require.
If you can fully anonymise your data and avoid the need
to store any personal information, do so. If you must store
personal data, consider the security of data not just when it
is stored but when it enters, moves within and leaves your
environment. Examine every point where personal data could
be read by an employee or malicious third party and ensure
you have pursued every measure within your control to protect
it. Delete data when it has been processed according to its
agreed purpose.
Verify that automated systems meet GDPR
stipulations
In addition to standardising data handling requirements and
penalties for misuse, the GDPR introduced considerations that
can impact AI solutions:
• Article 22 (Paragraph 1): “The data subject shall have
the right not to be subject to a decision based solely on
automated processing, including profiling, which produces
legal effects concerning him or her or similarly significantly
affects him or her.”
• Article 22 (Paragraph 2): “Paragraph 1 shall not apply
if the decision:
»
is necessary for entering into, or performance of,
a contract…[or]
»
is based on the data subject’s explicit consent.”
• Article 22 (Paragraph 3): “In the cases referred to in
[Paragraph 2], the data controller shall implement suitable
measures to safeguard the data subject’s rights and
freedoms and legitimate interests, at least the right to obtain
human intervention on the part of the controller, to express
his or her point of view and to contest the decision … and,
in Recital 71 only, the right to an explanation of the decision.”
These articles are yet to be comprehensively tested in court.
However, they explicitly prohibit legal effects – such as
sentencing and parole decisions – that result solely from
automated processing undertaken without the individual’s
explicit consent, when consent is required.
What constitutes “similarly significant” effects, “explicit
consent” (beyond acceptance of an extensive set of online
conditions containing a relevant paragraph) and whether
something is “necessary” to perform a contract are subject to
interpretation at this time.
If you are developing an AI system that could materially impact
an individual’s life, therefore, it is prudent to consider making
your system advisory only and including a human check.
Once case law has better established the meanings of the
terms above, there will be greater clarity regarding the
implications of the legislation.
The GDPR and Convention 108 impose
obligations of explainability
Article 22 (Paragraph 3) of the GDPR, which requires
companies to protect the data they control and allows
individuals to challenge an automated system they believe
is treating them unfairly, demands a robust explanatory
framework for the outputs of your systems. Convention 108 of
the Council of Europe (https://bit.ly/2n6POrT), adopted into
UK and EU law in May 2018, imposes related requirements:
• Convention 108 (Article 8): “Every individual shall have a
right… to obtain, on request, knowledge of the reasoning
underlying data processing where the results of such
processing are applied to him or her.”
Convention 108 affords individuals the right to understand
how decisions made about them, using data processing,
have been developed. Because every individual possesses
this right, you must be able to explain, in lay terms, how
decisions that affect individuals are made.
Explore varying approaches to explainability
Beyond the stipulations of Convention 108 of the Council
of Europe, there is growing demand more broadly for greater
explainability of AI systems. Improved explainability was a
recommendation, for example, from the UK Parliamentary
Science and Technology Select Committee on AI.
Regulatory or pragmatic demands may force you to
consider the explainability of your systems.
For a system that uses a decision tree, it will be
straightforward to explain how data maps to the system’s
decision. For machine learning-based systems, and particularly
deep learning-based systems, this will not be possible.
There may be thousands of abstract numbers corresponding
to the connections in the network that contribute to its output.
These numbers will be meaningless to individuals who seek
to understand the system, which is why many AI systems are
considered to be ‘black box’ and inexplicable.
There are, however, means of explanation that do not involve a
system’s mathematics. These approaches consider the impact
of variables inputted to a system and their influence on the
output. There are several techniques you can apply including
Inferred Explanation, Feature Extrapolation and Key Variable
Analysis (Fig. 41). Which you favour will depend on the level
of explainability you require and the challenge of providing it.
Consider the security of data not
just when it is stored but when it
enters, moves within and leaves
your environment.
Source: MMC Ventures
Fig. 41. Three approaches to explainability
Approach
Inferred Explanation
Feature Extrapolation
Key Variable Analysis
Very high
Very slow
Thorough
Challenging to understand
Low
Fast
Easy to understand
Limited explanatory power
Moderate
Slow
Easy to understand
Limited applicability
Difficulty
Speed
Advantages
Disadvantages
81
80
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 6
Regulation & Ethics
1. Inferred Explanation: Inferred Explanation is the easiest
way to explain AI. The algorithm is not described and
a ‘black box’ is retained around it. Correlations are
considered between system inputs and outputs, without
explaining the steps between.
By demonstrating examples of decisions, individuals can see
the correlations between input data and output decisions
(Fig. 42), without detail regarding how the inputs and outputs
are connected. Inferred explanation does not provide
complete clarity regarding a model, but will demonstrate
how decisions relate to inputs in a manner that will be
satisfactory in many situations.
2. Feature Extrapolation: Some systems, including financial
models and systems that materially impact individuals,
require an explanation – beyond correlation – of how
models reach their conclusions. While more effort, it is
possible to evaluate features in data that are activating
parts of a network. This is particularly fruitful for image
classification systems. Using test data, and reversing the
flow of data in a network, you can create images that
demonstrate the features that activate a particular layer in
the network (Fig. 43). Further, Google recently released
a library for TensorFlow to undertake this visualisation
automatically, within a browser, during training
(bitly.com/2R6XeZu). While not suitable for all AI systems,
feature extrapolation provides a degree of explainability in
a manner that non-technical individuals can appreciate.
Source: MMC Ventures
Fig. 42. Inferred Explanation
Variables A and C lead to decision 3
AI
Fig. X: Title TBC
PURCHASE
PATTERN A
PURCHASE
PATTERN C
DECISION
3
PURCHASE
PATTERN B
DECISION
2
DECISION
1
Source: Zeiler and Fergus, https://bit.ly/2JjF4R0
Fig. 43. Feature Extrapolation
Determining parts of an image critical to network output
There are means of explanation
that do not involve a system’s
mathematics.
Some systems require an explanation
– beyond correlation – of how models
reach their conclusions.
It is possible to evaluate
features in data that are
activating parts of a network.
83
82
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 6
Regulation & Ethics
3. Key Variable Analysis: If you wish to provide the most
precise explanation of your system, you must analyse the
impact of each input on the system’s decision-making
process and overall decision. This will require a full
statistical analysis and is a significant undertaking. For each
output decision, you will require an example of input data
that strongly results in that decision. Change each variable,
in turn, from the value that leads to Decision 1 to the value
that leads to Decision 2. Then, change the variables in
combination. The effort required will increase exponentially
according to the number of variables you have, and the
process will be time-consuming. However, you will be
able to determine whether any system inputs, singularly
or in combination, have a disproportionate effect on your
system’s output decision (“variable bias”). You may find,
for example, that your model places a high importance
on gender when providing an output decision. This is
possible, even if gender is not explicit in your data, if you
have other closely associated variables (such as vocation
in a gender biased industry).
Key variable analysis has drawbacks as well as advantages.
In addition to being complex and resource-intensive, it can be
difficult to explain results accessibly. Further, if you explain your
model in a high degree of detail, malicious third parties can
use this information to force results from your model that they
wish to see.
Develop a framework for ethical data use
When developing and deploying AI systems, as well as
providing sufficient explainability it is important to use data
ethically. “Plan for ethical AI from the outset and underpinning
all initiatives. It’s got to be foundational, not an afterthought”
(Steven Roberts, Barclays). In addition to the intrinsic
importance of doing so, a growing number of companies are
incurring reputational and financial costs from failing to do so.
The Durham Police Constabulary, in conjunction with
computer science academics, is trialling a framework –
ALGOCARE – to ensure its AI system uses data transparently
within an explainable, ethical process. Many companies with
AI systems also have frameworks in place, albeit privately and
often loosely defined. While every company’s framework
differs, ALGOCARE highlights issues you should consider
when managing data.
• Advisory: Is the output of the algorithm used as a
suggestion or a fact? How to interpret the output of a
system is a key consideration. Does a human, or automated
system, act on the output without further thought or
investigation? Do you wish your car, for example, to brake
automatically if danger is perceived (even when the system
is incorrect) or to warn you so you can make the decision?
To an extent, what is optimal may be determined by the
domain of the problem.

Too often, the numbers returned alongside a label are
interpreted and used as a confidence score. Usually,
however, they will be a probability distribution that has
been tuned to give an output above a specific level for
the predicted result. Even incorrect results can have high
probabilities. Decisions based on the “confidence” of
the network decision, therefore, can be disastrous. While
there are tuning techniques that better align a network’s
prediction probabilities with confidence levels (Guo et al,
https://bit.ly/2JiRNTS) they are rarely used given the time
required and teams’ focus on measures of accuracy.

• Lawful: Is your data gathered lawfully and do you have the
right to use it? Under the GDPR, individuals may not have
consented to their data being processed in the way you
intend. Data gathered without the informed consent of the
individual should not be used.
• Granularity: Do you understand the meaning of the data
you feed into your model? To avoid biased results and
models that fail when exposed to real-world data, it is
important to do so. What variables are missing, combined
or inferred? How varied is your data? Do you have sufficient
time series data? Data scientists can excel in this regard,
questioning data and anticipating problems before
building models.
• Ownership: Who owns the intellectual property associated
with the data and algorithms you use? Many companies
use academic data sets and models to validate concepts
in the early stages of development. Confirm that the
licenses, for both the data and models you use, are suitable
for commercial use. The availability of something on the
internet does not confer on your company the right to use it.
• Challengeable: Do the outcomes of your system need to
be available to individuals? For example, under GDPR may
the system be challenged? In some sectors, there will be
a greater need than others to be open about the basis of
your results. If you undertake all projects assuming that your
results will be challenged, you will be prepared.
Source: MMC Ventures
Fig. 44. How to select an approach to explainability
Use this approach if you:
Inferred Explanation
Feature Extraction
Key Variable Analysis
– Seek a high-level overview of your AI system
– Believe correlation offers sufficient explainability
– Require detail from within the network
– Have a network type (e.g. images) where abstractions can
be mapped onto input data
– Require detail about the importance of variables
– Seek to prevent unwanted bias in your variables
– Require detail regarding how variables lead to decisions
– Have limited time
– Require precise impact of input variables, not general
features
– Are not using an assignment–based or generative
AI network
– Have limited time
– Seek to publish your results
– Wish to offer a layperson’s guide to your model
Avoid this approach if you:
If you wish to provide the most precise
explanation of your system, you must
analyse the impact of each input on
the system’s decision-making process
and overall.
“Plan for ethical AI from the
outset and underpinning
all initiatives. It’s got to
be foundational, not an
afterthought.”
Steven Roberts, Barclays
85
84
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 6
Regulation & Ethics
• Accuracy: How is your system tested? How is changing
data or inaccuracies fed back into the system? Is dated data
removed? Many companies report ‘accuracy’ by cherry-
picking precision or recall (Chapter 5), which will overstate
model performance. Continuous testing with real world
data is the only way to verify your model’s performance.
• Responsible: Are ethical considerations deliberated
and stated? The impact of your system will depend on
its domain and the consequences of false positives and
false negatives. If your system could adversely impact
an individual’s life, you must understand and consider
the implications.
• Explainable: Has your model been explained, in jargon-
free language, to as great an extent as possible without
exposing your intellectual property? Explanations that avoid
technical terminology will enable everyone in your business
to understand what you are creating and to challenge it.
“Having a framework to explain your models is valuable for
all stakeholders” (Dr Janet Bastiman, Chief Science Officer,
StoryStream). What does your model do? How do inputs
correlate with outputs?
EU and UK Parliamentary committees, including the Science
and Technology Select Committee and the House of Lords
Artificial Intelligence Select Committee on AI, are engaged on
issues of AI, explainability and data privacy. The UK Science
and Technology Select Committee, for example, launched
an inquiry into the use of algorithms in public and business
decision-making. Further, more specific, legislation is
probable. Ensure that a senior member of your team (Chief
Science Officer, Head of AI or Head of Data) is responsible
for staying up-to-date regarding proposed legislation and
the impact it could have on your business.
EU and UK Parliamentary
committees are engaged on the
issues of AI and explainability.
Ensure that a senior member of your team is responsible
for staying up-to-date regarding proposed legislation.
87
86
The AI Playbook
The step-by-step guide to taking advantage of AI in your business
Chapter 6
Regulation & Ethics
This report is intended for general public guidance and to highlight issues. It is not intended to apply to specific circumstances or to
constitute financial, investment or legal advice. Barclays, MMC and their affiliates, directors, employees and/or agents expressly disclaim any
and all liability relating to or resulting from the use of all or any part of this report or any of the information contained herein.
No representation or warranty, express or implied, is given by or on behalf of MMC or Barclays as to the accuracy, reliability or completeness
of the information or opinions contained in this report. The report contains estimates and opinions which are not a reliable indicator of future
performance and may prove to be incorrect. MMC and Barclays accept no responsibility for updating the report for events or circumstances
that occur subsequent to such dates or to update or keep current any of the information contained herein.
This report is not and should not be taken as a financial promotion or investment advice. MMC, Barclays and their affiliates, directors, and
employees may have investments, trading positions or advisory relationships with the companies mentioned in the report. Readers should
always seek their own legal, financial and tax advice before deciding whether to make any investments.
Barclays Bank UK PLC. Authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential
Regulation Authority (Financial Services Register No. 759676). Registered in England.
Registered no. 9740322 Registered Office: 1 Churchill Place, London E14 5HP.
MMC Ventures
2 Kensington Square
London W8 5EP
www.mmcventures.com
Barclays UK (BUK) Ventures
Registered office:
1 Churchill Place
London
E14 5HP
www.home.barclays
Report design by
Aviary Creative
www.aviarycreative.co.uk
88
The AI Playbook
The step-by-step guide to taking advantage of AI in your business