Non-linear regression is a type of statistical analysis used to model a relationship between independent variables (predictors) and a dependent variable (outcome) when that relationship cannot be represented by a straight line. Instead of fitting a simple straight line, these algorithms fit a curved line to the data. 🤖

The core idea is to find a function that best describes the pattern in the data, allowing for curves, bends, and other complex shapes. This is different from linear regression, which assumes the relationship is linear (a straight line).

# --- 1. Import Necessary Libraries ---

import numpy as np

import matplotlib.pyplot as plt

from sklearn.tree import DecisionTreeRegressor

from sklearn.svm import SVR

from sklearn.preprocessing import PolynomialFeatures, StandardScaler

from sklearn.linear_model import LinearRegression

from sklearn.pipeline import make_pipeline

# --- 2. Generate Non-Linear Sample Data ---

# Let's create data that follows a quadratic curve (y = ax^2 + bx + c) plus some noise.

# A simple straight line will not fit this data well.

np.random.seed(42)

X = np.sort(10 * np.random.rand(80, 1), axis=0)

y = np.sin(X).ravel() + np.random.randn(80) * 0.1

# --- 3. Polynomial Regression ---

# This method transforms the input features into polynomial features (e.g., x^2, x^3)

# and then fits a linear regression model to these new features.

print("--- Training Polynomial Regression Model ---")

# Create a pipeline that first creates polynomial features (degree=4),

# then fits a linear regression model.

poly_model = make_pipeline(PolynomialFeatures(degree=4), LinearRegression())

poly_model.fit(X, y)

y_poly_pred = poly_model.predict(X)

# --- 4. Decision Tree Regressor ---

# This model learns a set of if-then-else rules, creating a step-like function.

# It's excellent at capturing complex, non-linear patterns without assuming any specific

# mathematical shape for the relationship.

print("\n--- Training Decision Tree Regressor Model ---")

# Create and train the model. `max_depth` controls the complexity of the tree.

tree_model = DecisionTreeRegressor(max_depth=4)

tree_model.fit(X, y)

y_tree_pred = tree_model.predict(X)

# --- 5. Support Vector Regressor (SVR) with RBF Kernel ---

# SVR can handle non-linear data by using a "kernel trick". The Radial Basis Function (RBF)

# kernel is a popular choice that can fit complex, smooth curves.

# Note: SVR is sensitive to the scale of the data, so we use StandardScaler.

print("\n--- Training Support Vector Regressor (SVR) Model ---")

# Create and train the SVR model. 'C' is a regularization parameter, and 'gamma'

# influences how far the influence of a single training example reaches.

svr_model = make_pipeline(StandardScaler(), SVR(kernel='rbf', C=100, gamma=0.1))

svr_model.fit(X, y)

y_svr_pred = svr_model.predict(X)

# --- 6. Visualize the Results ---

# Let's plot the original data and the line of best fit for each model.

plt.figure(figsize=(14, 8))

plt.scatter(X, y, s=20, edgecolor="black", c="darkorange", label="Actual Data")

# Plot the predictions

plt.plot(X, y_poly_pred, color="cornflowerblue", linewidth=2, label="Polynomial Regression (degree=4)")

plt.plot(X, y_tree_pred, color="yellowgreen", linewidth=2, label="Decision Tree (max_depth=4)")

plt.plot(X, y_svr_pred, color="navy", linewidth=2, label="SVR (RBF Kernel)")

# Add titles and labels

plt.title('Comparison of Non-Linear Regression Models', fontsize=16)

plt.xlabel('Feature (X)', fontsize=12)

plt.ylabel('Target (y)', fontsize=12)

plt.legend()

plt.grid(True)

plt.show()

# --- Predict a new value with each model ---

new_value = [[5.0]] # Predict the output for an input of 5.0

print(f"\n--- Predictions for new value X={new_value[0][0]} ---")

print(f"Polynomial Model Prediction: {poly_model.predict(new_value)[0]:.4f}")

print(f"Decision Tree Model Prediction: {tree_model.predict(new_value)[0]:.4f}")

print(f"SVR Model Prediction: {svr_model.predict(new_value)[0]:.4f}")

Key Characteristics

• Models Complex Relationships: They are essential when the underlying pattern in the data is not straightforward. For example, the relationship between advertising spend and sales might increase rapidly at first and then level off, a pattern that a straight line can't capture.

• Flexibility: These algorithms are highly flexible and can adapt to a wide variety of data shapes and patterns.

• Higher Accuracy (Potentially): When used on data with a genuine non-linear pattern, these models are far more accurate than linear models. However, they can also be more prone to overfitting, meaning they might learn the noise in the data instead of the underlying trend.

Common Non-Linear Regression Algorithms

Here are some of the most widely used non-linear regression algorithms:

1. Polynomial Regression

2. Support Vector Regression (SVR)

3. Decision Tree Regression

4. Random Forest Regression

5. Neural Network Regression