3  Intro to MLE

Maximum Likelihood Estimation

Note

We’ll spend two days on this topic!


Day 1

Settling In

  • Turn in PS0 at the front of the class
    • Is your name on front page?
    • Did you put a staple in top left corner?
  • Check in with the people at your table
  • Catch up on any recent Slack messages you might have missed
  • Review your notes from CP1: MLE and prepare to discuss



Syllabus Highlights

Quizzes and Exams

Frequency:

  • one quiz every 1–2 topics
  • typically on Fridays, approx. every other week, at least one week after you turn in the problem set(s) on that topic
  • dates: see Syllabus and Math/Stat 355 Deadlines GCal
    • conflict with one of these dates? contact me at least one week in advance

“Standards-Based” Structure:

  • two attempts on each concept
  • on first attempt:
    • pass = correct! –> you’ve done with this topic
    • revise = minor issue (eg algebra mistake) –> come to office hours to revise; if you do so correctly, score will be updated to pass
    • retake = more major issue –> try again on next quiz
  • on second attempt:
    • skip any questions about topics you passed or successfully revised on first attempt
    • complete any retakes
    • there will also be questions about new topics: everyone should try those!

Other Quiz Details:

  • closed-notes (but I’ll provide pdfs)
  • some questions about definitions/conceptual understanding, others will be derivations/proofs
  • I’ll write each portion (new, retakes) to take 30 mins or less

Final Exam:


Grading System

Details here: Syllabus - Course Details - Grading

To earn an A:

  • “consistently demonstrate a genuine effort to prepare”: complete almost all checkpoints, with most scores above ~80%
  • “consistently demonstrate a genuine effort to practice”: attend and actively participate in almost all classes, submit all problem sets and pass most
  • “consistently demonstrate an accurate understanding of course concepts”: pass almost all topics by your second attempt on quizzes
  • score at least 80% on the final exam

Coming soon: spreadsheets to help you track your feedback and progress


Attendance

You are warmly invited, encouraged, and expected to attend and participate in all class meetings.

There will almost certainly be days you are unable to attend! I particularly support and encourage you to do so if you are feeling sick or concerned about your well-being or safety.

Here’s what I expect if you miss class:

  • Check the schedule
  • Review the notes
  • Check in with a friend and/or come to office hours
  • Send me a quick email (no details needed!)


Questions?



Estimation

Goal: use observed data to estimate the unknown parameter(s) in a probability model

Techniques:

  • least squares
  • maximum likelihood estimation
  • method of moments
  • and more!



Checkpoint Discussion

NoteCheckpoint 1

CP1 covers the following content:

At your table, discuss:

  1. Let \(X_i\) represent the result of a coin flip with probability \(p\) of landing heads (\(X_i = 1\)) and probability \(1-p\) of landing tails (\(X_i = 0\)).

    1. If you observed \((X_1 = 0, X_2 = 1)\) and did not know the value of \(p\), would you be more willing to believe that \(p = 0.7\) or \(p = 0.4\)? Why? (HINT: You answered this on PS0!)

    2. What if \(p\) could be anywhere between 0 and 1? (HINT: We answered this in the MLE Intro video!)

    3. If we instead we observe \(n\) coin flips, the maximum likelihood estimate for \(p\) is \(\hat{p} = \frac{1}{n}\sum_{i=1}^n x_i\). (We derived this in the MLE Example video!) Explain why this estimate makes intuitive sense.

  2. What’s the goal / definition of (and intuition behind) Maximum Likelihood Estimation (MLE)?

  3. What are the usual steps to finding an MLE?

  4. How should we modify this process if we have two (or more) unknown parameters?

  5. Why do these steps typically involve taking a \(\log\)? Why are we allowed to do this?

Solutions

  1. …

    1. 0.4
    2. 0.5
    3. this is just the sample proportion!

  2. Find the value of the parameter that makes our observed data most likely. More specifically, the maximum likelihood estimator for an unknown parameter \(\theta\) is defined: \[\theta_e = \arg \max_\theta L(\theta),\] or equivalently, \(\theta_e\) is a maximum likelihood estimate for \(\theta\) if \[L(\theta_e) \ge L(\theta) \quad \forall \theta\]

  3. Steps for finding an MLE typically include:

    1. Find the likelihood: \(L(\theta) = \prod_{i=1}^n f_X(x_i; \theta)\), assuming \(X_1, ..., X_n\) are a random sample from a distribution with pdf \(f_X(x; \theta)\)
    2. Find the log-likelihood \(l(\theta) = \log\left(L(\theta)\right)\)
    3. Take the derivative of the log-likelihood with respect to the unknown parameter
    4. Set that derivative equal to zero and solve for the unknown parameter

  4. Get the partial derivatives with respect to each unknown parameter. Set those equal to zero, and solve the resulting system of equations.

  5. …

    • Why log? It’s often easier to work with the log-likelihood! Consider how many probability density/mass functions involve \(e^{...}\). We can also avoid the product rule since \[\log\left(\prod_{i=1}^n f_X(x_i; \theta)\right) = \sum_{i=1}^n \log\left(f_X(x_i; \theta)\right).\] (It’s easier to take the derivative across a sum than a product.)
    • Why allowed? This is allowed because log is a monotone increasing function. (Challenge: can you prove this?)



Practice Problems

Let’s do #1 together!

  1. Suppose \(X_1, ..., X_n\) are a random sample from the Normal pdf with parameters \(-\infty < \mu < \infty, \tau > 0\): \[f_X(x; \mu, \tau) = \sqrt{\frac{\tau}{2\pi}} e^{-\frac{\tau}{2}(x-\mu)^2}, \quad -\infty < x < \infty.\] Assume \(\tau\) is known. Find the MLE of \(\mu\).


Now, find some whiteboard space. Work on the remaining exercises in small groups from the boards.

  1. Repeat #2, assuming both \(\mu\) and \(\tau\) are unknown.

  2. Suppose \(X_1, ..., X_n \stackrel{iid}{\sim} \text{Uniform}(0, \theta)\). Find the MLE of \(\theta\).


Solutions

Check Slack for board notes!

  1. \(\hat\mu = \overline{X} = \frac{1}{n} \sum_{i=1}^n X_i\)

  2. \(\hat\mu = \overline{X}\) and \(\hat\tau = \left(\frac{1}{n}\sum_{i=1}^n (X_i - \bar{X})^2 \right)^{-1}\)

  3. This question is on your problem set this week. Hint: plot it!



Wrapping Up

After class today, please:

    • I recommend that you try problems 1, 2, 3, and 5 over the weekend. We’ll talk about 2 (Practice Problem 3, above!) and 5 (a slight variation on Practice Problem 2) in class on Monday.
    • If you finish all of those, try 4, 6, and 7. FYI: We’ll talk about the linear regression questions (6 and 7) in class on Wednesday.




Day 2

Settling In

  • Check out the finalized OH schedules: Kelsey | Nick
    • If my office hours don’t work for you, please post questions on Slack and/or set up a one-on-one appointment!
  • Check Slack for recent messages
  • Review your notes from last class and any practice problems you worked on over the weekend



Practice Problems (cont)

Let’s finish the practice problems from last class…

  • Finish #1 on the board, as a class.
  • Finish #2 in groups. Check your answer with Kelsey or Nick before moving on.
  • Attempt #3 in groups. We’ll pause to discuss as a class when everyone’s ready.
  • (If you’re done with all of the above, work on PS1!)



Notes: MLE

Goals

Estimation: use observed data to estimate the unknown parameter(s) in a probability model

Maximum Likelihood Estimation: find the value(s) of the unknown parameter(s) that makes our observed data as “likely” as possible


Definition

Version 1:

\(\theta_e\) is an maximum likelihood estimate for \(\theta\) if \[L(\theta_e) \ge L(\theta) \quad \forall \theta\]

Version 2:

\[\hat\theta = \arg \max_\theta L(\theta)\]


Usual Steps

  1. Find the likelihood \(L(\theta)\)

  2. Find the log-likelihood \(\log(L(\theta))\)

  3. Take the derivative of the log-likelihood with respect to \(\theta\)

  4. Set that derivative equal to zero and solve

  5. Recommended: pause and do an intuition check!

    • Are there restrictions on the values that \(\theta\) can take (eg must be positive, between 0 and 1)? If so, will your estimate respect those bounds?
    • What does the parameter you are trying to estimate represent? (eg is it the mean of the distribution? the variance? the maximum?) Then, does your estimator seem like a reasonable way to make a guess about that feature of the distribution based on the data you’ve collected?
    • NOTE: MLEs won’t always pass these checks, but they often do. If your answer doesn’t pass these checks it could be a sign that something went wrong in your calculations!


When the Usual Steps Don’t Work…

Scenario 1: The support of the distribution depends on the parameter we are trying to estimate. In this case, your MLE will usually be an order statistic (or a function thereof!).

WarningProbability Review: Order Statistics

Let \(X_1, X_2, ..., X_n\) be a random sample of size \(n\). Reorder the sample values from smallest to largest: \[X_{(1)} \leq X_{(2)} \leq \dots \leq X_{(n)}\] We call \(X_{(k)}\) the \(k\)th order statistic!


Examples:

The 1st order statistic, \(X_{(1)}\), is the sample minimum \[X_{(1)} = X_{\min} = \min\{X_1, X_2, ..., X_n\}.\]

The \(n\)th order statistic, \(X_{(1)}\), is the sample maximum \[X_{(n)} = X_{\max} = \max\{X_1, X_2, ..., X_n\}.\]

The median is also an order statistic:

\[ M = \begin{cases} X_{(n+1)/2} & \text{ if } n \text{ is odd } \\ \frac{1}{2} \left(X_{\frac{n}{2}} + X_{\frac{n}{2} + 1} \right) & \text{ if } n \text{ is even } \\ \end{cases} \]

Example: \(X \sim \text{Unif}(0, \theta)\)

Solution: look at likelihood (is it monotone increasing or decreasing? if so, check the endpoints)

Tips: pay attention to the support, draw a picture!


Scenario 2: We have a lot of unknown parameters and we can’t get a closed form solution to the system of equations we are trying to solve.

Example: logistic regression

Solution: numerical optimization techniques! (more on this in a future class)



What’s Next?

  • Our next few classes will focus on how we can use MLE in linear and logistic regression!

  • After class today, please: