{"id":1915,"date":"2019-05-14T12:00:00","date_gmt":"2019-05-14T10:00:00","guid":{"rendered":"https:\/\/kindsonthegenius.com\/blog\/basics-of-bayesian-inference\/"},"modified":"2026-07-05T03:23:13","modified_gmt":"2026-07-05T01:23:13","slug":"basics-of-bayesian-inference","status":"publish","type":"post","link":"https:\/\/kindsonthegenius.com\/blog\/basics-of-bayesian-inference\/","title":{"rendered":"Basics of Bayesian Inference"},"content":{"rendered":"<p>Bayesian Inference is simply a way of making statistical inference by applying <a href=\"https:\/\/kindsonthegenius.com\/tempsite\/machine-learning-101-rules-of-probability-bayes-theorem\/\">Bayes&#8217; Theorem.<\/a><\/p>\n<p>Assuming there is a particular hypothesis H.<\/p>\n<p>Let the probability of this hypothesis be p(H).<\/p>\n<p>According the Bayes Inference, we would update this probability are more information (or evidence) becomes available.<\/p>\n<p>To understand Bayes Inference, we need to briefly review Bayes&#8217; Theorem (or Bayes&#8217; Rule)<\/p>\n<p>&nbsp;<\/p>\n<h4><strong>Review of Bayes&#8217; Rule<\/strong><\/h4>\n<p>Just as explained the the previous lesson,\u00a0 we use Bayes&#8217; Theorem\u00a0 to find conditional probability.<\/p>\n<p>But in this discussion, we would say: Bayes&#8217; theorem is used to find posterior probability. This is explained earlier in <a href=\"https:\/\/kindsonthegenius.com\/tempsite\/machine-learning-101-application-of-bayes-theorem\/\">Application of Bayes&#8217; Theorem<\/a>.<\/p>\n<p>Posterior probability is derived from two things:<\/p>\n<ul>\n<li>prior probability<\/li>\n<li>likelihood function<\/li>\n<\/ul>\n<p>While prior probability has already been explained, likelihood function is deduced from the observed data.<\/p>\n<p>So let&#8217;s now state Bayes&#8217; Theorem<\/p>\n<p>&nbsp;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-832 aligncenter\" src=\"https:\/\/www.kindsonthegenius.com\/wp-content\/uploads\/2020\/09\/Bayes-Theorem-from-Exam-300x70.jpg\" alt=\"Bayes' Theorem\" width=\"300\" height=\"70\" \/><\/p>\n<p>In this equation:<\/p>\n<p>H is the hypothesis such that more information about it is expected. So the probability of H would be affected by new information or evidence, E.<\/p>\n<p>P(H) is called the prior probability. This is the initial probability of H before we receive the evidence or new information E.<\/p>\n<p>E is the evidence\u00a0 which means a new information received<\/p>\n<p>P(H | E) is the posterior probability of H given the new evidence E. Means the probability of H after E has been observed.<\/p>\n<p>P(E | H) similarly is the probability of E given H. This is called <strong><em>likelihood<\/em><\/strong>. Since the function of E with H is fixed, it therefore shows compatibility of the evidence E with the hypothesis H. The posterior probability is a function of H while the likelihood function is a function E.<\/p>\n<p>P(E) is the probability of E and is called the marginal likelihood. It is the same for all possible hypothesis being considered.<\/p>\n<p>&nbsp;<\/p>\n<h4><strong>Relationship between Prior and Posterior Probabilities<\/strong><\/h4>\n<p>Now, look at Bayes&#8217; theorem again:<\/p>\n<p>&nbsp;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-832 aligncenter\" src=\"https:\/\/www.kindsonthegenius.com\/wp-content\/uploads\/2020\/09\/Bayes-Theorem-from-Exam-300x70.jpg\" alt=\"Bayes' Theorem\" width=\"300\" height=\"70\" \/><\/p>\n<p>&nbsp;<\/p>\n<p>For the hypothesis H, only the factor in the numerator P(E | H) and P(H)\u00a0 actually affect the value of the posterior probability P(H | E).<\/p>\n<p>So you can see that the posterior probability is proportional to\u00a0 the likelihood P(E | H) and the prior probability P(H).<\/p>\n<p>Therefore, the posterior probability is proportional to the prior probability. If we rewrite Bayes&#8217; Theorem to reflect the proportion, we would have:<\/p>\n<p>&nbsp;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-833 aligncenter\" src=\"https:\/\/www.kindsonthegenius.com\/wp-content\/uploads\/2020\/09\/Posterior-Probability-is-Proportional-to-the-Prior-probability-300x67.jpg\" alt=\"\" width=\"300\" height=\"67\" \/><\/p>\n<p>&nbsp;<\/p>\n<p>In term of simple proportions we can write:<\/p>\n<p>&nbsp;<\/p>\n<h5 style=\"text-align: center;\"><em>P(H | E) = k. P(H)<\/em><\/h5>\n<p>&nbsp;<\/p>\n<p>This means that the factor k is would be given by:<\/p>\n<p>&nbsp;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-834 aligncenter\" src=\"https:\/\/www.kindsonthegenius.com\/wp-content\/uploads\/2020\/09\/Proportionality-of-posterior-probability-300x103.jpg\" alt=\"\" width=\"168\" height=\"58\" \/><\/p>\n<p>&nbsp;<\/p>\n<p>This factor is known as the impact of E on the probability of H. That is what effect the new evidence E have on the probability of H,\u00a0 p(H)<\/p>\n<p>&nbsp;<\/p>\n<h4><strong>Formal Definition of Bayesian Inference<\/strong><\/h4>\n<p>Now that you understand Bayes&#8217; Theorem, let&#8217;s now define Bayesian Inference.<\/p>\n<p>Let\u00a0<span class=\"mwe-math-element\"><img decoding=\"async\" class=\"mwe-math-fallback-image-inline\" src=\"https:\/\/wikimedia.org\/api\/rest_v1\/media\/math\/render\/svg\/4f5c5435030c952a58a756e691ea64f60c1bd240\" alt=\"{\\tilde {x}}\" aria-hidden=\"true\" \/><\/span>\u00a0 be a data point<\/p>\n<p>Let\u00a0\u03b8 be a parameter of the data point&#8217;s distribution. That is\u00a0 x ~ p(x |\u00a0\u03b8)<\/p>\n<p>Let\u00a0\u03b1 b a <em>hyperparameter<\/em> of the parameter distribution. That is\u00a0\u03b8 ~ p(\u03b8 |\u00a0\u03b1 )<\/p>\n<p>Let X be the sample, that is a set of n observed data points, x<sub>1<\/sub>, x<sub>2<\/sub>,&#8230;, xn<\/p>\n<p>Now\u00a0<span class=\"mwe-math-element\"><img decoding=\"async\" class=\"mwe-math-fallback-image-inline\" src=\"https:\/\/wikimedia.org\/api\/rest_v1\/media\/math\/render\/svg\/4f5c5435030c952a58a756e691ea64f60c1bd240\" alt=\"{\\tilde {x}}\" aria-hidden=\"true\" \/>\u00a0 is the new data point we need to predict the distribution<\/span><\/p>\n<p>&nbsp;<\/p>\n<p><strong>Bayesian Inference States that:<\/strong><\/p>\n<ul>\n<li>the prior distribution is the distribution of the parameters prior to observation of any data. That is p(\u03b8 |\u00a0\u03b1)<\/li>\n<li>the sampling distribution is the distribution of the observed data based on its parameters. That is p(X |\u00a0\u03b8). This is the likelihood. It can also be written in terms of L(\u03b8 | X)<\/li>\n<li>the marginal likelihood\u00a0 is therefore the distribution of the observed data over the parameters.<\/li>\n<li>similarly, the posterior distribution is the distribution of the parameters after the data is observed. This is of course the Bayes&#8217; rule you know<\/li>\n<\/ul>\n<p>The derivation if Bayes Inference is given below:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-835 \" src=\"https:\/\/www.kindsonthegenius.com\/wp-content\/uploads\/2020\/09\/Bayesian-Inference.jpg\" alt=\"Bayesian Inference\" width=\"490\" height=\"219\" \/><\/p>\n<p>We can state this as:<\/p>\n<p>&#8216;posterior probability equals likelihood times prior over evidence&#8217; or<\/p>\n<p>&#8216;posterior probability is proportional to likelihood times prior&#8217;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Bayesian Inference is simply a way of making statistical inference by applying Bayes&#8217; Theorem. Assuming there is a particular hypothesis H. Let the probability of &hellip; <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"pagelayer_contact_templates":[],"_pagelayer_content":"","footnotes":""},"categories":[414],"tags":[],"class_list":["post-1915","post","type-post","status-publish","format-standard","hentry","category-programming"],"_links":{"self":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1915","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/comments?post=1915"}],"version-history":[{"count":1,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1915\/revisions"}],"predecessor-version":[{"id":2083,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1915\/revisions\/2083"}],"wp:attachment":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/media?parent=1915"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/categories?post=1915"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/tags?post=1915"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}