Building Machine Learning from the Ground Up: Developer Recreates Scikit-Learn Algorithms in Pure Python with Surprising Results

By Tech Insights Desk
Published: October 2023

Main Facts

In an ambitious display of foundational programming and algorithmic comprehension, software developer Hariom Lohar has successfully built four core machine learning models completely from scratch. Using only the Python standard library—eschewing powerful data science frameworks like NumPy, SciPy, and Pandas—Lohar implemented Logistic Regression, Decision Trees, Random Forests, and K-Means clustering.

Dubbed minisklearn, the open-source repository offers a transparent look into the mechanics of popular machine learning algorithms. To test his hand-rolled models, Lohar deployed them on the classic Titanic dataset, comprising 889 rows and 7 features, with a survival rate of approximately 38%. Utilizing identical train-test splits (test_ratio=0.2, seed=0) and custom-built confusion-matrix evaluation metrics, Lohar compared his raw Python implementations against industry-standard libraries.

The results yielded both expected challenges and an astonishing revelation: while Lohar’s from-scratch classifiers (Logistic Regression, Decision Trees, and Random Forests) struggled with recall and lagged behind scikit-learn‘s optimized performance, his custom implementation of K-Means clustering matched the industry standard’s inertia to the exact decimal.

The project, now publicly available on GitHub under the repository name mini-sklearn, has sparked discussions within the developer community regarding the nuances of algorithmic implementation, feature scaling, numerical optimization, and the deep comprehension gained by coding foundational models without dependencies.

Chronology: The Journey of minisklearn

Phase 1: Conceptualization and the Standard Library Constraint

The genesis of the project stemmed from a desire to move beyond abstract API calls (model.fit() and model.predict()) and understand the underlying mathematics and logic driving modern machine learning. Rather than relying on vectorized array operations powered by NumPy and C-optimized libraries, Lohar restricted himself strictly to Python’s built-in data structures (such as lists and dictionaries) and the standard library.

This constraint meant writing custom matrix multiplication logic, sorting algorithms for split points, and distance metrics from scratch. The primary goal was educational: to peel back the layers of abstraction that shield modern developers from the underlying mechanics of data science.

Phase 2: Dataset Preparation and Custom Metric Engineering

To establish a rigorous benchmark, Lohar selected the Titanic survival prediction dataset—a staple binary classification problem featuring a mix of categorical and continuous variables. To ensure a fair comparison, the dataset was subjected to a rigid preprocessing pipeline, maintaining a consistent test ratio of 0.2 and a random seed of 0.

Crucially, Lohar did not stop at writing the models; he also manually hand-rolled evaluation metrics. Instead of importing accuracy, precision, recall, and F1 scores from sklearn.metrics, he built custom confusion-matrix functions from scratch to evaluate True Positives, False Positives, True Negatives, and False Negatives directly from raw prediction lists.

Phase 3: Training and the Unexpected Benchmarks

When the models were finally trained and evaluated against scikit-learn, a distinct divergence emerged. The classification models—Logistic Regression, Decision Trees, and Random Forests—exhibited a systematic conservative bias, drastically under-predicting the positive class ("survived").

Conversely, when Lohar ran his custom K-Means clustering algorithm, the results defied expectations. The calculated inertia—the sum of squared distances of samples to their closest cluster center—matched scikit-learn‘s output to the absolute decimal. This confirmed that Lohar’s implementation of Lloyd’s algorithm for centroid initialization and updates was mathematically and logically pristine.

Supporting Data: A Side-by-Side Performance Breakdown

To understand the exact performance delta between minisklearn and scikit-learn, Lohar documented the quantitative metrics across identical test splits. The comparison highlights both the triumphs of pure-Python implementation and the pitfalls of lacking advanced numerical optimizations.

Performance Metrics Comparison Table

Analyzing the Classifier Lag

The data reveals a clear and consistent pattern: while Lohar’s classifiers achieved respectable baseline accuracies (hovering around 68% to 70%), their recall metrics collapsed. While scikit-learn models achieved recall scores between 0.70 and 0.79, the minisklearn classifiers struggled within the 0.29 to 0.44 range.

Lohar diagnosed the root causes of this discrepancy as technical artifacts of pure-Python implementations without advanced data preprocessing:

1. Unscaled Features and Learning Rates: In Logistic Regression, gradient descent struggles significantly when features are unscaled, compounded by a learning rate that may have been too conservative or poorly tuned for raw numerical ranges.
2. Exact ID3 Splits on Continuous Data: Tree-based models (Decision Trees and Random Forests) rely heavily on finding optimal split thresholds. Without optimized sorting and heuristic shortcuts used in production libraries, the hand-rolled ID3 splitting logic fragmented continuous inputs inefficiently, leading to over-pruned or poorly generalized decision boundaries that defaulted to predicting the majority class ("did not survive").

Technical Insights and Developer Reflections

Building machine learning libraries from scratch exposes the invisible heavy lifting performed by frameworks like scikit-learn, NumPy, and PyTorch. Lohar’s project sheds light on several crucial engineering realizations:

The Perfection of Lloyd’s Algorithm in K-Means

The absolute parity in K-Means inertia (742,754.6) is a testament to the deterministic nature of distance calculations and cluster assignments when logic is sound. Unlike gradient-based optimization—which is sensitive to step sizes, convergence tolerances, and floating-point optimizations—K-Means relies on straightforward Euclidean distance formulas and iterative mean updates. Reaching the exact decimal without vectorization proves that Lohar successfully mapped the fundamental mechanics of unsupervised clustering loops.

The Illusion of Simplicity in Classifiers

Conversely, the performance gap in classifiers illustrates why modern data science relies on heavily optimized libraries. Implementing gradient descent in pure Python means iterating over large lists rather than leveraging vectorized C arrays. Furthermore, handling edge cases such as zero-division in precision metrics, categorical variable encoding, and numerical stability in sigmoid functions requires meticulous engineering that libraries have spent decades refining.

Implications for the Broader Developer Community

Lohar’s experiment resonates far beyond a personal portfolio project, touching upon broader philosophies of software engineering, artificial intelligence education, and modern development workflows.

Moving Beyond the "Neural Network First" Mentality

One of the most poignant takeaways articulated by Lohar involves the modern rush toward deep learning. In an era where developers routinely throw heavy neural networks or complex gradient boosting frameworks at simple tabular problems, projects like minisklearn serve as a vital reminder of the power of classical algorithms.

Lohar noted that in future projects, he intends to prioritize classical models and simpler architectures where appropriate, rather than reflexively jumping to deep neural networks. Classical models are often more interpretable, computationally cheaper, and surprisingly resilient in specialized domains where deep learning models can overfit or behave unpredictably.

The Value of First-Principles Learning

For junior developers and data science students, the project provides a blueprint for mastering machine learning. Relying entirely on high-level APIs can create an illusion of competence. By stripping away dependencies and forcing oneself to write loss functions, tree splitters, and distance matrices using native language constructs, engineers transition from mere users of tools to true builders of technology.

As open-source repositories like mini-sklearn continue to gain visibility on platforms like GitHub and DEV Community, they inspire a return to fundamentals—proving that deep technical curiosity remains one of the most powerful drivers of software engineering excellence.