Pre-training techniques are very important for creating Large Language Models (LLMs). These techniques help models understand language by learning from large amounts of text data. We’ll talk about two main techniques: self-supervised learning and masked language modeling. Let’s understand them!

1. Self-Supervised Learning

Self-supervised learning is a method where the model learns from the data itself without needing labeled data. The idea is to create pseudo-labels from the data and use them for training.

Key Ideas:

• Predictive Learning: The model learns to predict parts of the data from other parts.
• Data Augmentation: Creating different versions of the data to give the model more examples to learn from.

Example: Predicting the Next Word Imagine a sentence: “The cat sat on the ___.” The model tries to predict the next word (“mat”) based on the context.

Code Example:

from transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch

# Load pre-trained model and tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")

# Encode input text
input_text = "The cat sat on the"
input_ids = tokenizer.encode(input_text, return_tensors="pt")

# Predict next word
outputs = model(input_ids)
logits = outputs.logits

# Decode prediction
predicted_id = torch.argmax(logits[:, -1, :], dim=-1).item()
predicted_word = tokenizer.decode(predicted_id)

print(f"Input: {input_text}")
print(f"Predicted next word: {predicted_word}")

Output:

Input: The cat sat on the
Predicted next word: mat

2. Masked Language Modeling (MLM)

Masked Language Modeling is a special type of self-supervised learning used in models like BERT (Bidirectional Encoder Representations from Transformers). The model learns to predict missing (masked) words in a sentence.

Key Ideas:

• Masking: Randomly hide some words in a sentence.
• Prediction: The model tries to predict the masked words based on the context.

Example: Masking a Word Given the sentence: “The cat sat on the mat,” we might mask the word “cat” to get “The [MASK] sat on the mat.”

Code Example:

from transformers import BertTokenizer, BertForMaskedLM
import torch

# Load pre-trained model and tokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertForMaskedLM.from_pretrained("bert-base-uncased")

# Encode input text with a masked token
input_text = "The cat sat on the mat"
masked_text = "The [MASK] sat on the mat"
input_ids = tokenizer.encode(masked_text, return_tensors="pt")

# Predict masked word
outputs = model(input_ids)
logits = outputs.logits

# Find the masked token position
masked_index = (input_ids == tokenizer.mask_token_id).nonzero(as_tuple=True)[1].item()

# Decode prediction
predicted_id = torch.argmax(logits[0, masked_index]).item()
predicted_word = tokenizer.decode(predicted_id)

print(f"Input: {masked_text}")
print(f"Predicted masked word: {predicted_word}")

Output:

Input: The [MASK] sat on the mat
Predicted masked word: cat

Summary

• Self-Supervised Learning: The model learns by predicting parts of the data from other parts, using the data itself as a guide.
  • Example: Predicting the next word in a sentence.
  • Code: Using GPT-2 to predict the next word.
• Masked Language Modeling (MLM): The model learns to fill in the blanks (masked words) in a sentence.
  • Example: Predicting a masked word in a sentence.
  • Code: Using BERT to predict the masked word.

These techniques are the foundation for training large language models, allowing them to learn from vast amounts of text data without needing a lot of labeled data.