{"id":186,"date":"2018-01-12T10:04:00","date_gmt":"2018-01-12T09:04:00","guid":{"rendered":"https:\/\/kindsonthegenius.com\/blog\/2018\/01\/12\/maximum-likelihood-estimation-mle-in-machine-learning\/"},"modified":"2020-08-22T09:26:59","modified_gmt":"2020-08-22T07:26:59","slug":"maximum-likelihood-estimation-mle-in-machine-learning","status":"publish","type":"post","link":"https:\/\/kindsonthegenius.com\/blog\/maximum-likelihood-estimation-mle-in-machine-learning\/","title":{"rendered":"Maximum Likelihood Estimation (MLE) in Machine Learning"},"content":{"rendered":"
\n
What is Maximum Likelihood(ML)? and What is Maximum Likelihood Estimation (MLE)?
These is a very important concept in Machine Learning and that is what we are going to cover today. <\/p>\n
<\/a><\/div>\n

<\/ins>Table of Content<\/b><\/span><\/span><\/p>\n

    \n
  1. What is Maximum Likelihood Estimation(MLE)?<\/span><\/li>\n
  2. Properties of Likelihood Extimates <\/span><\/li>\n
  3. Deriving the Likelihood Function <\/span><\/li>\n
  4. Log Likelihood<\/span><\/li>\n
  5. Applications of MLE <\/span><\/li>\n
  6. Final Thoughts<\/span><\/li>\n<\/ol>\n

    <\/span> <\/span>1. What is Maximum Likelihood Estimation?<\/b><\/span><\/span>
    The likelihood of a given set of observations is the probability of obtaining that particular set of data, given chosen probability distribution model.
    MLE is carried out by writing an expression known as the Likelihood function for  a set of observations. This expression contains an unknown parameter, say, \u03b8  of he model. We obtain the value of this parameter that maximizes the likelihood of the observations. This value is called maximum likelihood estimate.<\/span>
    Think of MLE as opposite of probability. While probability function tries to determine the probability of the parameters for a given sample, likelihood tries to determine the probability of the samples given  the parameter.<\/span><\/p>\n

    <\/ins>2. Properties of Maximum Likelihood Estimates<\/b><\/span><\/span>
    MLE has the very desirable properties especially for very large sample sizes some of which are:
    likelihood function are very efficient in testing hypothesis about models and parameters
    they become unbiased minimum variance estimator with increasing sample size
    they have approximate normal distributions<\/p>\n

    3. Deriving the Likelihood Function<\/b><\/span><\/span>
    Assuming a random sample x1<\/sub>, x2<\/sub>, x3<\/sub>, … ,xn<\/sub><\/span><\/i><\/span><\/span> which have joint probability density and denoted by:<\/p>\n

    L(\u03b8<\/span>)<\/i><\/span> = f(x1<\/sub>, x2<\/sub>, x3<\/sub>, … ,xn<\/sub>|<\/span><\/i>\u03b8)<\/span><\/i><\/span><\/span><\/span><\/div>\n


    <\/p>\n

    where \u03b8 is a parameter of the distribution with unknown value.<\/span><\/div>\n
    We need to find the most likely value of the parameter \u03b8 given the set observations<\/span>. To do this, we use a likelihood function.<\/div>\n

    <\/p>\n

    The likelihood function is defined as:<\/span><\/div>\n
    L(\u03b8<\/span>)<\/i><\/span> = f(x1<\/sub>, x2<\/sub>, x3<\/sub>, … ,xn<\/sub>|<\/span><\/i>\u03b8)<\/span><\/i><\/span><\/span><\/span><\/div>\n
    which is considered as a function of <\/span>\u03b8 <\/span><\/div>\n
    If we assume that the sample is normally distributed, then we can define the likelihood estimate for <\/span>\u03b8 <\/span>as  the value of <\/span>\u03b8 that maximizes the L(<\/span>\u03b8), that is the value of <\/span>\u03b8 that makes the data set most likely.<\/span><\/div>\n
    We can split the function <\/span>f(x1<\/sub>, x2<\/sub>, x3<\/sub>, … ,xn<\/sub>|<\/span><\/i>\u03b8)<\/span><\/i><\/span><\/span><\/span> as a product of univariates such that:<\/span><\/div>\n
    L(\u03b8<\/span>)<\/i><\/span> = f(x1<\/sub>, x2<\/sub>, x3<\/sub>, … ,xn<\/sub>|<\/span><\/i>\u03b8)<\/span><\/i><\/span><\/span> = f(<\/span><\/span>x1<\/sub><\/span><\/i><\/span><\/span><\/span>|<\/span><\/i><\/span>\u03b8) +<\/span><\/span><\/span><\/span>f(<\/span><\/span><\/span><\/span><\/span><\/span>x2<\/sub><\/span><\/i><\/span><\/span><\/span><\/span><\/span><\/span><\/span>|<\/span><\/i><\/span><\/span><\/span>\u03b8), + <\/complete><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>f(<\/span><\/span><\/complete><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>x3<\/sub><\/span><\/i><\/span><\/span><\/span><\/complete><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>|<\/span><\/i><\/span><\/span><\/span>\u03b8) +… + <\/span><\/span><\/span><\/span><\/complete><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>f(<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>xn<\/sub><\/span><\/i><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>|<\/span><\/i><\/span><\/span><\/span>\u03b8) <\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/div>\n
    which would give us the same results.<\/div>\n

    <\/p>\n

    <\/a><\/div>\n

    So the question is ‘what would be the maximum value of <\/span><\/span>\u03b8 for the given observations? This can be found by maximizing this product using calculus methods, which is not covered in this lesson. <\/span><\/span><\/span><\/span>

    <\/span><\/span><\/span><\/span><\/p>\n

    <\/span><\/div>\n
    <\/span><\/ins>4. Log Likelihood<\/b><\/span><\/span><\/div>\n
    Maximizing the likelihood function derived above can be a complex operation. So to work around this, we can use the fact that the logarithm of a function is also an increasing function. So maximizing the logarithm of the likelihood function, would also be equivalent to maximizing the likelihood function. <\/div>\n
    This is given as:<\/div>\n
    <\/span><\/span><\/div>\n
    <\/a><\/div>\n
    So at this point, the result we have from maximizing this function is known as ‘maximum likelihood estimate<\/i>‘ for the given function<\/div>\n
    <\/span><\/span><\/div>\n
    5. Applications of Maximum Likelihood Estimation<\/b><\/span><\/span><\/div>\n
    MLE can be applied in different statistical models including linear and generalized linear models, exploratory and confirmatory analysis, communication system, econometrics and signal detection.<\/div>\n
    <\/div>\n
    6. Final Thoughts <\/b><\/span><\/span><\/div>\n
    Also it is important to note that calculating MLEs often requires specialized computer applications for solving complex non linear equations. However such tools are readily available.<\/div>\n
    I would recommend making some effort learning how to use your favorite maths\/analytics software package to handle and MLE problem
    <\/span><\/span><\/div>\n

    <\/div>\n<\/div>\n

    <\/ins> <\/ins><\/p>\n","protected":false},"excerpt":{"rendered":"

    What is Maximum Likelihood(ML)? and What is Maximum Likelihood Estimation (MLE)?These is a very important concept in Machine Learning and that is what we are … <\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0},"categories":[11,16],"tags":[],"_links":{"self":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/186"}],"collection":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/comments?post=186"}],"version-history":[{"count":1,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/186\/revisions"}],"predecessor-version":[{"id":1446,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/186\/revisions\/1446"}],"wp:attachment":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/media?parent=186"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/categories?post=186"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/tags?post=186"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}