Marketing has been one of the most significantly impacted functions by the rapid adoption
and advancement of Large Language Models (LLMs). These powerful Generative AI tools are
actively changing the way we approach various marketing tasks, from content creation and
copywriting to strategic planning and campaign development. However, despite the growing
interest and investment in LLMs, there has been a lack of comprehensive, unbiased
studies evaluating their performance against real-world marketing use cases.
To address this gap, our team of experienced marketers conducted the first impartial
study assessing the leading LLMs in the market. We recognized the need for a study that
was not only conducted for marketers by marketers but also one that focused on the
practical applications and challenges faced by marketing professionals in their daily
work.
Background
The global AI market has already been valued at $241.8 billion in 2023 as compared to
$135 billion in 2022, and it is expected to surge to a staggering $740 billion by 2030,
reflecting a CAGR of 17.3%, according to Statista. This surge is being spearheaded by
the United States, boasting the world’s strongest AI research capabilities,
followed closely by Europe. Enterprises are seeing a steep adoption rate. According to
OpenAI, 90% of Fortune 500 brands are actively using their tools and API internally.
Chart 1: Year-on-Year Growth
Chart 2: AI Research
Capabilities
Generative AI Adoption in Marketing
The marketing and advertising industry has emerged as a frontrunner in AI adoption, with
a remarkable 37% of professionals already utilizing this technology in their daily
tasks.
This rapid embrace is not surprising, considering the unique blend of creativity and data
analysis inherent to marketing. AI seamlessly integrates with this blend, offering
powerful tools for both creative exploration and data-driven decision-making.
Chart 3: Industry Comparison
In a 2023 study conducted with marketers in the United States, 73% of respondents
reported using generative artificial intelligence tools, such as chatbots, as a part of
their company’s work. This widespread adoption indicates a growing comfort level
and understanding of the potential benefits these tools offer.
Generative AI, particularly Large Language Models (LLMs), has revolutionized the way
people work across industries, providing innovative solutions to complex tasks. It has
found prominent usage in various job functions, notably in industries such as IT and
Technology, Finance, Healthcare, and Marketing.
Marketing Evaluation to Leading
LLMs Report
The study involved a panel of 20 seasoned marketers, each with a minimum of 6 years of
experience in the field. These experts were tasked with reviewing the output of
anonymized leading LLMs across a range of the most common marketing use cases, including
copywriting, marketing strategy development, content creation, creative ideation, and
campaign planning. To ensure an unbiased evaluation, we carefully crafted prompts that
were highly practical, realistic, and consistent across each model. This approach
guaranteed that no single model was favored or engineered to outperform others, a common
issue in LLM evaluations published by the creators of the models themselves.
The LLMs covered in the study:
- LLama 2 by Meta
- Gemini 1.5 by Google
- Perplexity by Perplexity
- ChatGPT (publicly available GPT 3.5) by OpenAI
- Claude 2.1 by Anthropic
- Claude 3 – Sonnet by Anthropic
By conducting this study, we aim to provide marketers with valuable insights into the
strengths and weaknesses of each LLM, empowering them to make informed decisions when
incorporating these tools into their marketing strategies and workflows. Our findings
shed light on the variability in performance across different aspects, such as
creativity, thoroughness, and coherence, highlighting the importance of selecting the
right LLM for specific marketing tasks. The data presented in this report demonstrates
the significant potential of AI applicability use case in marketing as the industry
continues to evolve.
The graphs below offer a broad overview of how various Large Language Models
(LLMs) performed across different marketing tasks. For a more in-depth analysis
and breakdown of these findings, download the free report by clicking the
download button below.
As the use of Generative Artificial Intelligence (Gen AI) continues to grow, a critical
question emerges: Which Gen AI tool is best suited for what purpose, and how do the
leading solutions compare? This report addresses this question by evaluating the
performance of six Large Language Models (LLMs) across various marketing tasks and
against diverse evaluation criteria.