{"id":1911,"date":"2019-04-20T12:00:00","date_gmt":"2019-04-20T10:00:00","guid":{"rendered":"https:\/\/kindsonthegenius.com\/blog\/machine-learning-101-basics-of-logistic-regression\/"},"modified":"2026-07-05T03:23:04","modified_gmt":"2026-07-05T01:23:04","slug":"machine-learning-101-basics-of-logistic-regression","status":"publish","type":"post","link":"https:\/\/kindsonthegenius.com\/blog\/machine-learning-101-basics-of-logistic-regression\/","title":{"rendered":"Machine Learning 101 \u2013 Basics of Logistic Regression"},"content":{"rendered":"

First I would like clarify that the Logistic Regression model is a model for classification<\/em>.<\/p>\n

Also note that Machine Learning 101 focuses on Supervised Learning. Therefore we always would be discussing Classification and Regression. Machine Learning 102 on the other hand, focuses on Unsupervised Learning (Clustering, Density Estimation and Dimensionality Reduction).<\/p>\n

 <\/p>\n

What is Logistic Regression?<\/strong><\/h4>\n

Logistic regression is a model used when the dependent variable follows a binomial distribution. Simply put, when the variable y, is binary.<\/p>\n

And although we would discuss details of probability distribution later, but just know that a distribution is what you get when you plot he probabilities of a random variable against the values of the random variable.<\/p>\n

So instead of saying<\/p>\n

y = f(x)<\/em><\/p>\n

we say:<\/p>\n

p(x) = f(x)<\/em><\/p>\n

Recall that in linear regression we used a model like this:<\/p>\n

p(X) =\u00a0\u00df0<\/sub> +\u00a0\u00df1<\/sub>x<\/em><\/p>\n

In case of logistic regression, we use the logistic function, given as:<\/p>\n

\"Logistic<\/p>\n

Don’t worry about how it this function looks. Its quite simple. It’s the same as:<\/p>\n

\"Logistic<\/p>\n

where<\/p>\n

Y =\u00a0\u00df0<\/sub> +\u00a0\u00df1<\/sub>X<\/em><\/p>\n

The logistic function is good for modelling binary response because, the output of this function would always be between 0 and 1 for all values of X.<\/p>\n

 <\/p>\n

Odds Ratio<\/strong><\/h4>\n

This is a very important concept you should know.<\/p>\n

We derive the odds ratio by modifying the logistic function. If we do that, we would have the function below:<\/p>\n

\"Odds<\/p>\n

So the quantity p(X)\/[1-p(X)] is called the odds. It can take values from 0 to infinity(\u221e)<\/p>\n

Value of odds close to 0 indicates very low probability while values close to\u00a0\u221e indicate very high probability.<\/p>\n

Odds are sometimes used instead of probability in certain fields. For instance in games.\u00a0 We can ask: what are the odds of winning this game?\u00a0 If the probability of winning is let’s say 0.9, then the odds of winning would be 9 – that is 9\/(1-9).<\/p>\n

Let’s go a little further.<\/p>\n

If we take the log of both sides of the odds equation, we would have the equation below:<\/p>\n

\"\"<\/p>\n

The log expression on the left-hand side is called the logit<\/em> or log-odds<\/em>. So we can see that the logistic regression model has a logit<\/em> that is linear in the variable X.<\/p>\n

 <\/p>\n

Unit Change in X<\/strong><\/p>\n

We want to understand the behavior of the logistic regression model. In Logistic regression, increasing value of X by on unit would change the odds by\u00a0\u03b21<\/sub> or similarly it multiplies the odds by e\u03b21<\/sub><\/sup>.<\/p>\n

However, since the relationship between p(X) and X is not linear,\u00a0\u03b21<\/sub> does not produce a corresponding change in p(X) associated with a unit increase in X.\u00a0 The amount that p(X) changes therefore depends on the current value of X.<\/p>\n

If\u00a0 \u03b21<\/sub>\u00a0 is positive, then increase in X will yield increase in p(X) but if\u00a0 \u03b21<\/sub>\u00a0 is negative, then increase in X would result in decrease in p(X). This is irrespective of the value of X.<\/p>\n","protected":false},"excerpt":{"rendered":"

First I would like clarify that the Logistic Regression model is a model for classification. Also note that Machine Learning 101 focuses on Supervised Learning. … <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"pagelayer_contact_templates":[],"_pagelayer_content":"","footnotes":""},"categories":[16],"tags":[],"class_list":["post-1911","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"_links":{"self":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1911","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/comments?post=1911"}],"version-history":[{"count":1,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1911\/revisions"}],"predecessor-version":[{"id":2079,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1911\/revisions\/2079"}],"wp:attachment":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/media?parent=1911"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/categories?post=1911"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/tags?post=1911"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}