So, your aim is to dive into AI studies? Yet you are unsure about how or where to begin?

In 2020, I shared a list of the Top 20 free Data Science, ML, and AI MOOCs available online. However, I have come to realize that enrolling in numerous courses is not the most effective approach.

To break free from the cycle of endless tutorials and truly gain expertise, you need to engage in practical applications, develop algorithms from scratch, implement research papers, and undertake engaging AI projects to solve real-world issues.

This piece endeavors to create a complimentary curriculum that aligns with this ideology. I am actively pursuing some of these courses, hence feel free to get in touch on Twitter or Linkedin if you wish to learn alongside me!

Moreover, please drop a comment if you identify any omissions!

Before delving into specifics, there are certain insights to consider regarding the curriculum and some guidance on the learning process.

Starting from the Top-down

This curriculum adopts a top-down methodology — initially focus on coding, theory later.

I prefer learning out of necessity. Whenever there is a problem to address, a solution to devise, or a prototype to construct, I will extensively search for the essential information, comprehend it, and subsequently act upon it.

For instance, my ambition is to evolve into an AI engineer well-versed in LLMs at a foundational level. This entails the ability to code transformers from the ground up and fine-tune LLMs on GPUs. Currently, I lack this proficiency due to gaps in my knowledge, which I am determined to bridge.

This curriculum primarily focuses on NLP; if you are interested in other AI specializations like computer vision or reinforcement learning, drop a comment below or directly message me on Twitter or Linkedin. I will provide you with some recommended resources.

Prior to inundating you with a plethora of links, I wish someone had emphasized two critical aspects I should have been aware of before embarking on the learning journey.

Educating in a Public Domain

The realm of knowledge is vast and continuous learning is imperative, especially in the field of AI where groundbreaking research and concepts emerge on a weekly basis.

The gravest mistake one can make is to confine learning to private spheres. By doing so, you limit opportunities for growth. Mere completion of tasks lacks significance. What holds greater value is how you assimilate the information, transform it into knowledge for dissemination, and conceive innovative solutions and ideas based on that knowledge.

Hence, it is vital to educate in a public space.

This can entail:

• drafting blogs and tutorials
• participating in hackathons and collaborating with peers
• asking and addressing queries in Discord communities
• embarking on projects of personal interest
• sharing newfound discoveries on Twitter

Moreover, when it comes to Twitter,

Employing Twitter as a Tool

When used effectively and in accordance with the right individuals, Twitter proves to be the most valuable social platform in contemporary times.

Whom to follow? Check out this curated AI list by Suhail.

Utilizing Twitter? Refer to Near’s guide on Effective Twitter Practices.

Reach out to individuals on Twitter. Be genuine, concise, and articulate your requirements clearly. This guide on Crafting Cold Emails by Sriram Krishnan is also applicable to Direct Messages.

Tweet Composition? Refer to Elements of a Tweet by Jason, the developer of Instructor, who accumulated 14k followers within months.

If you are perusing this article, do follow me on Twitter!

Contact me regarding your ongoing projects! Collaboration on intriguing endeavors always piques my interest.

Let’s now delve into the details.

Contents Overview

• Mathematical Foundations
• Recommended Tools
  ∘ Python
  ∘ PyTorch
• Machine Learning
  ∘ Creating Algorithms from Scratch
  ∘ Participation in Competitions
  ∘ Execution of Side Projects
  ∘ Deployment of Projects
  ∘ Supplementary Materials
• Deep Learning
  ∘ Fast.ai
  ∘ Involvement in More Competitions
  ∘ Implementation of Research Papers
  ∘ Exploring Computer Vision
  ∘ Diving into NLP
• Large Language Models
  ∘ Watching Neural Networks: Zero to Hero
  ∘ Participating in a Free LLM Boot Camp
  ∘ Creating Projects with LLMs
  ∘ Engaging in Hackathons
  ∘ Exploring Research Papers
  ∘ Developing Transformers from Scratch
  ∘ Suggested Blog Reads
  ∘ Viewing Umar Jamil’s Work
  ∘ Learning to Operate Open-source Models
  ∘ Prompt Engineering Techniques
  ∘ Fine-tuning LLMs
  ∘ Research about RAG
• Keeping Updated
• Alternative Curricula and Lists for Reference

Foundations in Mathematics

DALL·E

Machine learning heavily relies on three core mathematical pillars: linear algebra, calculus, as well as probability and statistics. Each of these elements plays an essential role in facilitating the optimal performance of algorithms.

• Linear Algebra: serves as the mathematical framework for data management and manipulation, with matrices and vectors acting as the primary medium for algorithmic interpretation and information processing
• Calculus: serves as the driving force for optimization in machine learning, enabling algorithms to learn and refine by comprehending gradients and rate variations
• Probability and Statistics: provide the fundamental principles for decision-making amidst uncertainty, enabling algorithms to forecast outcomes and learn from data through probabilistic and variance models

A phenomenal series on Math for Machine Learning from a developer’s viewpoint can be explored here by Weights & Biases.

source code)

If you prefer a code-centric approach to Linear Algebra, explore Computational Linear Algebra (video series, source code) shared by the developers of fast.ai.

Follow Introduction to Linear Algebra for Applied Machine Learning with Python in parallel with the course.

If you desire a more conventional approach, check out the lectures at Imperial College London — Linear Algebra & Multivariate Calculus.

View 3Blue1Brown’s Essence of Linear Algebra and Essence of Calculus.

Observe Statistics Fundamentals by StatQuest for statistics

Additional Resources

• Book: Mathematics for Machine Learning
• Paper: The Matrix Calculus You Need For Deep Learning

Resources

DALL·E

Python

Beginners can start here: Practical Python Programming.

If you are already adept in Python, consider Advanced Python Mastery

Both are exceptional courses from David Beazley, the creator of Python Cookbook.

Next, delve into James Powell’s presentations

Explore Python Design Patterns.

Additional Resources

• Book: Fluent Python, 2nd Edition (source code)
• Podcasts: Real Python & Talk Python

PyTorch

Watch PyTorch Tutorials by Aladdin Persson

The PyTorch website offers valuable content.

• PyTorch Examples
• Official PyTorch Tutorials
• FAQ page

Challenge yourself with some puzzles

• srush/Tensor-Puzzles: Solve puzzles. Enhance your PyTorch skills

Additional Resources

• Book: Programming PyTorch for Deep Learning

Machine Learning

DALL·E

Refer to the 100-page ML book.

Develop from Scratch

While studying, implement the algorithms from scratch.

Check out the repositories listed below

• eriklindernoren/ML-From-Scratch
• JeremyNixon/oracle
• trekhleb/homemade-machine-learning

For a challenging task, build PyTorch from scratch by following this course.

• MiniTorch: A DIY Course on Machine Learning Engineering (video series, source code)

Participate in Competitions

Apply your knowledge in competitive scenarios.

• Engage in ML competitions on platforms like bitgrit and Kaggle; find more in this article.
• Review prior winning solutions and analyze them

Engage in Side Projects

Explore Bringing machine learning to production by Vicki Boykis

She also documented her experience building Viberary, a semantic book search engine.

Obtain a dataset and develop a model (e.g., leverage earthaccess for NASA Earth data).

Construct a user interface using streamlit and share it on Twitter.

Deploy the Models

Put the models into production. Track your experiments. Learn to monitor models. Experience data and model drift firsthand.

Explore these valuable resources

• Made With ML
• DataTalksClub/mlops-zoomcamp: Free MLOps course
• chiphuyen/machine-learning-systems-design
• Evidently AI — ML system design: 300 case studies
• stas00/ml-engineering: Machine Learning Engineering Online Book

Additional Resources

• Machine Learning with PyTorch and Scikit-Learn (source code)
• [1811.12808] Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning
• Introduction to Machine Learning Interviews Book ·MLIB

Deep Learning

For those interested in a top-down approach, begin with fast.ai.

Fast.ai

• fast.ai (part1, part2) + W&B Study Group

If you enjoyed fast.ai, explore Full Stack Deep Learning.

For a more in-depth and conventional course, consider checking out UNIGE 14×050 — Deep Learning by François Fleuret.

If you need to delve into theory later on, these are excellent resources.

• Dive into Deep Learning (contains code samples in PyTorch, NumPy/MXNet, JAX, and TensorFlow)
• Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville
• Neural networks and deep learning
• Understanding Deep Learning (with practical notebooks)

Read The Little Book of Deep Learning on your mobile instead of scrolling through Twitter.

Read these while your neural networks are progressing.

• A Recipe for Training Neural Networks
• Deep Neural Nets: 33 years ago and 33 years from now

Engage in more competitions

• PlantTraits2024 — FGVC11 | Kaggle (computer vision)

Implement research papers

Explore labml.ai Annotated PyTorch Paper Implementations

Papers with Code serve as a valuable resource; check out BERT explained on their platform.

Below are some references for specific areas within Deep Learning

Computer Vision

Many individuals suggest the CS231n: Deep Learning for Computer Vision course. It’s demanding but rewarding if you persevere.

Reinforcement Learning

For RL enthusiasts, these resources are excellent:

• Spinning Up in Deep RL by OpenAI
• 🤗 Deep Reinforcement Learning Course — Hugging Face

NLP

Another standout Stanford course, CS 224N | Natural Language Processing with Deep Learning

Get familiar with Hugging Face: Hugging Face NLP Course

Explore this Super Duper NLP Repo

Informative articles and breakdowns

• BERT Research — Ep. 1 — Key Concepts & Sources · Chris McCormick
• The Illustrated Word2vec — Jay Alammar
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning
• Understanding LSTM Networks — colah’s blog
• PyTorch RNN from Scratch — Jake Tae

Additional resources

• Natural Language Processing with Transformers Book

Large Language Models

Start by watching [1hr Talk] Intro to Large Language Models by Andrej.

Then check out Large Language Models in Five Formulas, by Alexander Rush — Cornell Tech

Watch Neural Networks: Zero to Hero

It commences with elucidating and coding backpropagation from the ground up and culminates in developing GPT from scratch.

Neural Networks: Zero To Hero by Andrej Karpathy

Andrej recently shared a new video → Let’s build the GPT Tokenizer

You may also want to explore GPT in 60 Lines of NumPy | Jay Mody during this process.

Free LLM boot camp

A complimentary LLM Bootcamp released by Full Stack Deep Learning.

This boot camp covers prompt engineering, LLMOps, UX for LLMs, and guidance on launching an LLM app within an hour.

After completing this boot camp and feeling eager to build,

Develop with LLMs

Interested in creating applications with LLMs?

Watch Application Development using Large Language Models
by Andrew Ng

Read Building LLM applications for production by Huyen Chip

As well as Patterns for Building LLM-based Systems & Products by Eugene Yan

Consult the OpenAI Cookbook for practical guides.

Utilize Vercel AI templates to kickstart your projects.

Join hackathons

lablab.ai hosts fresh AI hackathons weekly. Feel free to reach out if you’re interested in teaming up!

If you wish to delve deeper into the theoretical aspects and grasp the functioning of everything:

Read papers

An exceptional articleby Sebastian Raschka on Understanding Large Language Models, presenting a compilation of papers you must peruse.

In a recent publication, he also shared another article featuring papers for your consideration in January 2024, focusing on mistral models.

Explore his substack called Ahead of AI.

Creating Transformers from the Ground Up.

Peruse The Transformer Family Version 2.0 | Lil’Log for a concise summary.

Select the preferred format and establish it from the start.

Documentation

• Attention Is All You Need
• The Illustrated Transformer
• The Annotated Transformer by Harvard
• Thinking like Transformer

Articles

• Creating a Transformer From Scratch — Part One: The Attention Mechanism (part 2) (code)
• Understanding and Coding the Self-Attention Mechanism of Large Language Models From Scratch by Sebastian Raschka, PhD
• Transformers from scratch

Visuals

• Coding a Transformer from scratch on PyTorch, with full explanation, training and inference
• NLP: Implementing BERT and Transformers from Scratch

You are now equipped to build transformers from scratch. Nonetheless, more awaits.

View these Stanford CS25 — Transformers United videos.

Insightful Blog Posts

• Gradient Descent into Madness — Building an LLM from scratch
• The Illustrated Transformer — Jay Alammar
• Some Intuition on Attention and the Transformer by Eugene Yan
• Speeding up the GPT — KV cache | Becoming The Unbeatable
• Beyond Self-Attention: How a Small Language Model Predicts the Next Token
• Llama from scratch (or how to implement a paper without crying) | Brian Kitano
• Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch

View Umar Jamil

He offers detailed video insights on papers, including accompanying code demonstrations.

• LoRA: Low-Rank Adaptation of Large Language Models — Explained visually + PyTorch code from scratch
• Mistral / Mixtral Explained: Sliding Window Attention, Sparse Mixture of Experts, Rolling Buffer
• Attention is all you need (Transformer) — Model explanation (including math), Inference and Training
• LLaMA explained: KV-Cache, Rotary Positional Embedding, RMS Norm, Grouped Query Attention, SwiGLU
• Retrieval Augmented Generation (RAG) Explained: Embedding, Sentence BERT, Vector Database (HNSW)

Here are additional resources related to LLMs that are by no means exhaustive. Refer to the LLM Syllabus for a detailed overview of LLMs.

Mastering the Execution of Open-Source Models.

Utilize ollama: Get up and running with Llama 2, Mistral, and other large language models locally

Python & JavaScript Libraries were recently launched here

Engaging in Prompt Engineering

Consult Prompt Engineering | Lil’Log

ChatGPT Prompt Engineering for Developers by Ise Fulford (OpenAI) and Andrew Ng

Delve into other brief courses available for free at DeepLearning.ai.

Enhancing LLMs through Fine-Tuning

Refer to the Hugging Face fine-tuning guide.

Helpful guidance: Fine-Tuning — The GenAI Guidebook

Discover axolotl.

Benefit from this informative article: Fine-tune a Mistral-7b model with Direct Preference Optimization | by Maxime Labonne

RAG

An exceptional piece by Anyscale: Building RAG-based LLM Applications for Production

An in-depth analysis of Retrieval Augmented Generation by Aman Chadha

AI Education Resources

• The Batch | DeepLearning.AI | AI News & Insights
• Deep Learning Weekly
• Interconnects | Nathan Lambert
• AI Tidbits | Sahar Mor

Other Curriculums/Listicles to Explore

While my list is not exhaustive, if you desire more resources, here are a few suggestions.

• openai/syllabus.md
• AI Canon | Andreessen Horowitz
• AI Learning Curation — LLM Utils
• Threshold to the AI Multiverse | Open DeepLearning
• louisfb01/start-llms: A complete guide to start and improve your LLM skills in 2023

I trust this will aid you on your AI expedition!