{"id":1903,"date":"2019-04-13T12:00:00","date_gmt":"2019-04-13T10:00:00","guid":{"rendered":"https:\/\/kindsonthegenius.com\/blog\/machine-learning-101-rules-of-probability-bayes-theorem\/"},"modified":"2026-07-05T03:22:47","modified_gmt":"2026-07-05T01:22:47","slug":"machine-learning-101-rules-of-probability-bayes-theorem","status":"publish","type":"post","link":"https:\/\/kindsonthegenius.com\/blog\/machine-learning-101-rules-of-probability-bayes-theorem\/","title":{"rendered":"Machine Learning 101 \u2013 Rules of Probability & Bayes\u2019 Theorem"},"content":{"rendered":"

We will now consider some of the important rules of probability. Meanwhile we would also understand the meaning of terms along the line. They include:<\/p>\n

    \n
  1. Conditional Probability<\/a><\/li>\n
  2. Sum Rule<\/a><\/li>\n
  3. Product Rule<\/a><\/li>\n
  4. Bayes Theorem<\/a><\/li>\n
  5. Summary<\/a><\/li>\n<\/ol>\n

    Some term you need to know includes Joint Probability<\/em> and Marginal Probability<\/em><\/p>\n

    Let’s start with Conditional Probability.<\/p>\n

     <\/p>\n

    1. Conditional Probability<\/strong><\/h4>\n

    We would use the example of the box of apples and oranges from Lecture 8 (Introduction to Probability Theory)<\/a>.\u00a0 And we would illustrate by an example.<\/p>\n

    \"\"<\/p>\n

    Assume that we randomly select a box. Then from this box, we randomly pick a fruit without replacing the first one. If we already know the probability of picking a box to be P(B), then what is the probability P(F) that the fruit is an apple.<\/p>\n

    In other words, given a known probability, P(B), what is the probability of P(F).<\/p>\n

    This is written of the form:<\/p>\n

    P(F | B)<\/p>\n

    and read as: the conditional probability of F, given B.<\/p>\n

     <\/p>\n

     <\/p>\n

    2. The Sum Rule<\/strong><\/h4>\n

    In this case, we would state the Sum Rule and then explain it. Later, we would apply it to our apple and oranges example.<\/p>\n

    Note that I’m using upper and lower case p<\/em> the same.<\/p>\n

    \"Sum<\/p>\n

    We would have written it in terms of the boxes example but it would be clearer we understand the formula.<\/p>\n

    P(X, Y) is known as the joint probability<\/em> <\/strong>of X and Y. It is read as the probability of X and Y.<\/p>\n

    Also, P(X) is known as marginal probability of X.<\/p>\n

    Therefore, the sum rule simply means that we can find the probability of X by summing up all the joint probabilities of X over Y. But you may ask, how do we find the joint probability?<\/p>\n

    We get it using the product rule!<\/p>\n

     <\/p>\n

     <\/p>\n

    3. The Product Rule<\/strong><\/h4>\n

    As mentioned, the product rule helps us find joint probability. The product rule states:<\/p>\n

    \"Product<\/p>\n

    What if we interchange X and Y? The know about the symmetry property which says that the product rule is same as:<\/p>\n

    \"Symmetry<\/p>\n

    Here P(Y, X) is the joint probability of Y and X while P(X | Y) is the conditional probability of X\u00a0 given Y.<\/p>\n

    Finally, P(X) is the marginal probability<\/strong><\/em> of X (or just the probability of X<\/em>).\u00a0 But you may ask: how do we find conditional probability?<\/p>\n

    We get it using the Bayes’ Theorem!<\/p>\n

     <\/p>\n

     <\/p>\n

    4. Bayes Theorem<\/strong><\/h4>\n

    Bayes theorem helps us find conditional probability.\u00a0 It simply derived from the product rule.<\/p>\n

    If we rewrite the product rule\u00a0 in terms of P(X|Y)\u00a0 we would have:<\/p>\n

    \"Bayes<\/p>\n

    Now we can use the symmetry property from the product rule to replace the numerator. The we have:<\/p>\n

    \"Final<\/p>\n

    This is the legendary Bayes’ theorem!<\/p>\n

    I would recommend you take some time to get it around your head. Maybe, write it out a number of times. Also see how you can derive it.<\/p>\n

     <\/p>\n

    5. Summary<\/strong><\/h4>\n

    What have we learnt so far?<\/p>\n

      \n
    • First you now understand the terms, conditional probability, marginal probability and joint probability<\/li>\n
    • You now know of the sum rule which helps us find the marginal probability. It states that the marginal probability of X is the sum of joint probabilities of X and Y over Y<\/li>\n
    • You also now know of the product rule which helps us find the joint probability<\/li>\n
    • You also know know of Bayes’ theorem which helps us find the conditional probabilities.<\/li>\n<\/ul>\n

      In the next class, we would see how we can apply all of these to solve a problem<\/p>\n","protected":false},"excerpt":{"rendered":"

      We will now consider some of the important rules of probability. Meanwhile we would also understand the meaning of terms along the line. They include: … <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"pagelayer_contact_templates":[],"_pagelayer_content":"","footnotes":""},"categories":[16],"tags":[],"class_list":["post-1903","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"_links":{"self":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1903","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/comments?post=1903"}],"version-history":[{"count":1,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1903\/revisions"}],"predecessor-version":[{"id":2071,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1903\/revisions\/2071"}],"wp:attachment":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/media?parent=1903"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/categories?post=1903"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/tags?post=1903"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}