One Thing all AI Breakthroughs have in common

By

Piotr Grudzień

January 18, 2023

•

5

min read

AI Breakthroughs

Artificial Intelligence has come a long way in the past decade. We have seen breakthroughs in computer vision, natural language processing, and game playing, to name a few. One of the common factors among these breakthroughs is the use of large amounts of accurately labeled data.

In this blog post, we will take a closer look at some of the recent AI breakthroughs and how self-supervised learning has played a significant role in these advancements. We will also explore how the field is moving towards an all-embracing “look at history and predict the future” setup, and how that shift is impacting business use cases of Machine Learning.

‍

2012 - The Krizhevsky, Sutskever, Hinton ImageNet breakthrough

Sample from the ImageNet dataset (source)

One of the earliest breakthroughs in AI was the 2012 ImageNet moment, where a deep neural network called AlexNet achieved a top-5 error rate of 15.3% (more then 10 percentage points improvement over previous results) on a dataset of over 14 million images.

It was a game-changer in the field of computer vision, as it showed that deep learning could vastly outperform traditional image recognition methods. It also led to widespread adoption of deep learning in computer vision tasks, and the ImageNet dataset remains a benchmark for image classification models to this day.

Today, you can access the 14,197,122 manually labelled images on the dataset’s website.

‍

2013 - The DeepMind Atari breakthrough (game self-play)

Breakout screenshot from the Deepmind Atari paper

In 2013, DeepMind made a breakthrough in game playing with their AI agent that learned to play Atari games using reinforcement learning, a type of unsupervised learning.

‍

2016 - The DeepMind AlphaGo breakthrough (game self-play)

The next game DeepMind went after was Go

In 2016, AlphaGo, another game-playing AI agent developed by DeepMind, achieved a breakthrough by using a combination of supervised and unsupervised learning. In March 2016, AlphaGo beat Lee Sedol (world’s top Go player) 4:1 in an uprecedented event - often compared to the chess match between Deep Blue and Garry Kasparov in 1997.

‍

2020 - GPT-3 (scrape the internet and predict the next word / token)

In 2020, GPT-3, a language model developed by OpenAI, made a breakthrough by using unsupervised learning on huge amounts of text data scraped from the Internet. The model was trained on the task of predicting the next word or token in a sentence. GPT-3 is based on the transformer architecture. It can generate very coherent and human-sounding text. GPT-3 was trained on a massive dataset of unlabelled data and has been used as a base model to create other models like ChatGPT.

‍

2022 - DALL-E-2, Stable Diffusion (scrape the internet for image – alt-text pairs)

Image has been generated with Stable Diffusion 2.1 using prompt: ‘Futuristic humanoid robot in front of a waterfall taking off using a jet pack, fireworks in the background, high-resolution photograph with great lighting’

In 2022, DALL-E-2, another model developed by OpenAI, made a breakthrough by using self-supervised learning on image <> alt text pairs scraped from the Internet. Stable Diffusion (try it out here), an open-source model followed soon after.

‍

Next AI breakthrough?

Self-supervised learning has been a significant factor in recent AI advancements. Crucially, it allows AI models to learn from large amounts of unlabeled data, which can be obtained orders of magnitude faster than labelled data (compare game self-play or scraping text off the Internet to manually annotating text or images).

That is particularly relevant in today’s digital world, packed full of vast amounts of data in the form of videos (think of TikTok, Netflix, GIFs). It must be that the next breakthrough comes from a predict what happens next in this video setup.

‍

It all comes down to data

All of the above seem to be converging to an all-embracing look at history and predict the future setup. We’ve been observing this shift in business use cases of Machine Learning for a while now:

online advertisers accumulate user profiles rather than do one-off predictions of click-through rates,
the same thing happens with Machine Learning for transaction fraud detection: all users get profiled over time based on their financial data,
Spotify, YouTube, Netflix, etc. will show you recommendations that fit how your profile has been evolving and lead to where you want to go next rather than those that maximise the immediate click probability.

It all comes down to generating a huge amount of accurately labelled data. And there’s no greater dataset than all of what has happened so far and no more accurate label than seeing what happened next.

‍

Piotr Grudzień

Similar articles

Artificial Intelligence

LLM benchmarks: What are they and can you trust them?

Zuzanna Dubanowska

Artificial Intelligence

What are AI Hallucinations? It’s a feature, not a bug

"What are they, what causes them, and what can you do to prevent that problem... but is it really a problem?"

Zuzanna Dubanowska

Artificial Intelligence

AIaaS: The What, Why, And Wow of AI as a Service

We're witnessing the rise of AIaaS — Artificial Intelligence as a Service. Learn what AIaaS is and why should you care.

Zuzanna Dubanowska

February 23, 2024

Artificial Intelligence

RAG vs Fine-tuning for your business? Here's what you need to know

Considering custom AI solutions for your business? Discover how Retrieval Augmented Generation (RAG) offers a flexible and affordable alternative to fine-tuning LLMs.

Zuzanna Dubanowska

February 22, 2024

Artificial Intelligence

AI Chatbot Buyer Guide: 6 crucial factors to consider

Looking for the perfect AI chatbot for your business? Read this and learn about the 6 unobvious factors that you should pay attention to.

January 31, 2024

Artificial Intelligence

Is ChatGPT a real polyglot? Problems with multilingualism of LLMs

Let's talk about the limitations of Large Language Models in terms of their multilingual capabilities, why things are the way they are, and what can you do about it.

Zuzanna Dubanowska

January 23, 2024

Artificial Intelligence

GPT Tokens Explained - what they are and how they work

Large language models generate text token by token. But what are tokens exactly? And how do they relate to entropy and data compression?

Piotr Grudzień

October 6, 2023

Artificial Intelligence

GPT-4: How Multimodal Learning Takes Us Closer to Human-level Performance

Let's investigate how the integration of multi-modality in the newly released GPT-4 will enhance its knowledge and capabilities.

Maciej Kowalski

September 29, 2023

Artificial Intelligence

One Thing all AI Breakthroughs have in common

All recent AI breakthroughs had one thing in common: a clever way of generating a huge amount of accurately labelled data.

Piotr Grudzień

September 29, 2023

Artificial Intelligence

ChatGPT: what it is, use cases and limitations

In today’s blog post we will introduce you to ChatGPT - the latest project published by OpenAI. We will explain what it is, suggest some use cases but also mention important limitations.

September 29, 2023

Artificial Intelligence

3 Hard Truths about Generative AI

At Quickchat AI, we think and talk about Generative AI a lot. There are some things that rarely get mentioned in popular content on the topic.

Piotr Grudzień

September 29, 2023

Succeed in the new world redefined by AI.
Don't get left behind.

Start building for free