A Review of Machine Learning and Reinforcement Learning.
Over the past two years, I’ve been taking courses at Georgia Tech while working full-time. During this time, I was traveling, consulting, and working at a startup. If you have a similar workload or situation that requires 40–50 hours’ worth of full-time work a week, this may help you. This blog reviews popular courses for anyone in the OMSCS program with the Machine Learning specialization.
Let’s start with Reinforcement Learning.
CS7642: Reinforcement Learning and Decision Making
Out of all the courses that I’ve taken during this program, Reinforcement Learning has been the most engaging. It’s a machine learning course at the graduate level, so the expectation is that you have some experience in machine learning. It’s not the end of the world if you don’t. CS7641 Machine Learning would be the most helpful for easing the learning curve. But you have to start somewhere, and it’s not by any means an easy course.
The course consists of three components:
Homework (6) 30% — Submitted as a quiz via Canvas.
Projects (3): 45% — Replicating results from recent relevant research papers.
Exams (1) 25% — Free form answers.
I describe this course as abstract because all the projects don’t have a public rubric. The majority of your grade comes from your written analysis. This course depends on your ability to explain the fundamentals of reinforcement learning. The first half of the course covers Markov Decision Problems, Value/policy Iteration, and Q-learning. Then the course material builds on these concepts to discuss partially observable MDPs and game theory.
Time & Effort
I found myself spending 35–45 hours a week on this course. I took this course during the summer, and it was a bit of a time crunch working on the projects and homework. Gathering data and writing took as much time as designing and replicating the research papers assigned for each project. While you have to write and read research papers, I found this course had very attentive TAs to guide you through understanding and to structure your papers. If you don’t have too much experience reading research papers or academic text, you will spend another 5 hours a week on the course. Expect to find yourself participating in office hour discussions or reviewing course material.
I found myself leveraging the recommended resources and text suggested by Charles Isbell, the course instructor. He knows what he’s doing, and there was quite a generous curve for the course, which amounted to about 10–15% from what I could surmise. (I would say the goal is to stay at or above the averages on all assignments).
Charles Isbell and Michael Littman made the lectures truly enjoyable. Some parts were quite difficult to wrap my head around, so I recommend watching the lectures every weekend if possible. All in all, the mathematical proofs don’t matter for the final exam, but you want to ensure you understand or make a note of the main outcomes of each proof and lecture to do well on the final exam.
All in all, I found this course engaging. The course staff was impressive, invested and knowledgeable in the field. You don’t always get a hand full of Ph.D. students and researchers from OpenAI wanting to support your growth and learning in the space. I also thought the projects were clever and open-ended, allowing you to explore different libraries and languages. E.g., Pytorch, Tensorflow, etc., in Python.
CS7641: Machine Learning
This course was textbook Machine Learning material. The course covered approaches and techniques in supervised machine learning, unsupervised machine learning, and reinforcement learning. There was less work for this course, four graded assignments, a midterm, and a final exam.
Assignments(4) — 50%
Midterm — 25%
Final — 25%
Like Reinforcement Learning, your assignment grade comes mostly from your research paper report and discussion of the work. You won’t be given a rubric for your research paper but expect to be graded on how you describe and layout your analysis.
In this course, you can leverage code samples from the internet. For example, taking code snippets from tutorials sklearn. The assignments were all based on two datasets, which you can generate at random or borrow from dataset repositories. I used the Pima Diabetes dataset and Madelon dataset from UCI (https://archive.ics.uci.edu/ml/index.php).
Luckily most of the assignments were straightforward and experiment-based. I found myself working overnight on the weekends to capture data from experiments and structure them into graphs and tables. If you’re not familiar with Latex, I would recommend getting an editor like Word or Google Docs to help you write your report in IEEE format or something reasonable. The average grades for all assignments were in the 50–60% range. You can do well following research paper structure and ensuring you give context to your work for assignments.
Taking CS7646 Machine Learning for Trading helpful in understanding common data structures for ML like decision trees and ensemble-based learners. Taking CS7642 Reinforcement Learning was also useful because the final assignment for Machine learning focuses on q-learning vs. policy and value iteration, which are concepts that were covered in reinforcement learning. I leveraged my knowledge in the space to breeze through the last part of the course and used that time to double down on studying for the final exam.
Time and Effort
The final and midterm exams were similar in structure to reinforcement learning. Meaning you have the chance to lose points for any question. Because of the open-ended nature of the exams, you’ll want to ensure you can fully explain your answers and discuss topics related to machine learning.
Again, as with reinforcement learning, there is a curve, and you’ll want to stay near or above the average scores for the course to get your target grade. I spent about 25–35 hours a week on this course. There was less material to get through compared to reinformcement learning, but I spent more time on the assignments plotting graphs and modifying my experiments to ensure I was replicating research results properly.
Machine Learning Specialization at Georgia Tech
If you are specializing in Machine Learning for the graduate computer science program at Georgia Tech, you’ll run into both the Machine Learning and Reinforcement Learning courses. Both courses were challenging, so I shared some of experience with both courses in the program. If this was helpful and you’d like to read computation photography and computer vision review, I’ll be happy to write one once the spring 2021 semester is over. Cheers to everyone pursuing their master’s degree! Best of luck.