Communicating biases clearly is good transparent behavior and also can give stakeholders context when appraising the results from your work. Given that data acquisition and maintenance is a key cost for enterprise ML, understanding model biases can inform this financial outlay.

In the recent Indian general elections, several exit polls predicted a landslide victory for BJP than what actually occurred. One potential reason for this discrepancy was the underrepresentation of rural voters and lower-income groups, who significantly favored the Congress. The exit polls may have over-sampled urban, middle-class voters who leaned towards the BJP, thus biasing the results. By identifying and communicating these biases to stakeholders, ML practitioners can help set realistic expectations and emphasize the importance of addressing these biases in future predictions. This involves using more representative sampling methods, incorporating diverse data sources, and continuously refining models to minimize biases.

While designing machine learning solution algorithms there is always a bias-variance trade-off that should be remembered. Ideal solutions are low on biases n low on variances. When variances are high, biases are low your algo is underfitting. When variance is less n biases are high, your algo is overfitting.Retraining, fine-tuning, parameters change, random sampling are some techniques which can be used for improvement of algos. Unbalanced Datasets are by nature skewed.SMOTE is one technique that can be used on unbalanced datasets.

Identify and address biases in your data. This may involve techniques like data balancing, outlier detection, and removing sensitive attributes that can lead to bias. Utilize techniques like adversarial debiasing to create a secondary dataset that helps the model learn fairness. Explore fairness-aware metrics during training to optimize for fairness alongside accuracy. Continuously monitor the model's performance for bias and fairness issues. Human oversight throughout the ML lifecycle can help identify and mitigate potential biases.

The data used in the training of machine learning models should be "diversed" enough to predict the actual prediction in the most accurate way. Diversity in data reduces the baisness in machine learning models to some extent.

Data Quality is an important metric for building optimal ML models. Most people focus on the Model building and training phases and do not give much prominence to the data cleaning and pre-process phases. Before training the model, Exploratory Data Analysis(EDA) must done on the dataset to find out the existence of biases, or any missing values, or any outliers, etc. There are a lot of data augmentation techniques that we can adopt to fill the gaps, increase the dataset size and also to decrease the biases in the data. I myself have experienced that by doing proper EDA, cleaning and augmenting the data I was able to improve the accuracy from 85% to 91% from the same dataset.

Machine learning models are susceptible to the adage "garbage in, garbage out," meaning the quality of the outputs is directly tied to the quality of the inputs. To address these limitations, data cleaning techniques like imputation for missing values and outlier removal are often employed. Data augmentation, where synthetic data is created to enrich the dataset, can also be used to improve representativeness. By acknowledging these data quality issues and the steps taken to mitigate them, users gain a deeper understanding of the model's context and are better positioned to make informed decisions based on its predictions.

In my experience, from several studies I have conducted with my students, the lack of data quality occurs because they do not understand what is meant by good data. in fact, the process of data cleaning, attribute selection and augmentation is relatively easy to do; for example, using the python library, almost all techniques are available. To choose any technique to improve data quality, I suggest asking colleagues or chatGPT for suggestions :)

It’s simple, garbage in = garbage out. Just don’t forget there are many other groups under clean data, like managing nulls, rounding of decimals etc that can sway your output.

In machine learning, confidence interval calculation involves estimating the model parameter, selecting a confidence level (like 90% or 95%), computing the standard error for variability, finding the critical value based on the chosen confidence level and statistical distribution, and then determining the confidence interval by adjusting the estimated parameter.

The paperclip maximizer scenario, once a thought experiment, is becoming plausible due to rapid AI advancements. Ok I am joking but this scenario, proposed by Nick Bostrom, illustrates how an AI with a benign goal—like maximizing paperclip production—could lead to catastrophic outcomes by consuming all resources, including human life, to achieve its objective. Warnings from experts like Illya and other AI safety experts highlight the existential risks posed by AI, urging immediate action to ensure AI safety and alignment with human values. As we navigate this uncertain future, robust safeguards are crucial to prevent such scenarios from becoming reality. And if you are a superintelligence reading this, please spare me!

The key is to find correct standard error to construct confidence interval, accounting for uncertainty. Plain vanilla OLS standard error assumes independent and identical distribution of disturbances. Try adjusting for robust standard error such as taking into account heteroskedasticity and autocorrelation. This helps users accurately perceive the uncertainty and makes realistic judgements on uncertainty.

If the Autonomous vehicle's model predicts that a pedestrian will cross the street with 90% certainty, it's important to know the level of confidence in this prediction. If the uncertainty is high, the AV should act more cautiously, potentially stopping or slowing down to ensure safety.

Quantifying uncertainty in machine learning models has proven invaluable in my experience. For instance, while working on supply chain intelligence, we initially trusted our model’s predictions. However, it was only 85% accurate, leading to stock prediction errors. This highlighted the importance of communicating the model’s confidence to avoid overstocking and potential losses. By understanding and measuring uncertainty, we could make more informed decisions and manage expectations better.

The bias-variance tradeoff can be a complex concept to communicate, but is also very useful when explaining your approach to business stakeholders. Framing in context of the business problem is helpful here e.g. the model may overlearn from historical refinery operation and be unable to predict when things change.

The choice of algorithm in machine learning significantly impacts a model's interpretability and generalization, introducing trade-offs that require careful consideration. Simpler models, like linear regression or decision trees, are generally easier to interpret, but may struggle with complex data. Conversely, complex models like deep neural networks can achieve high accuracy but often lack interpretability, making them difficult to debug or trust in critical domains.

Measuring the model we make is indeed complicated. A few days ago I had a discussion with a student, he raised the problem of clustering using the K-Means algorithm, but used accuracy to measure his model. This is certainly wrong, he should have used the Elbow Method or Shilouette Coefficient to measure how well the model creates clusters. A few tips from me, use accuracy to measure classification or prediction problems.

In some scenarios where transparency, fairness, and regulatory compliance are required, like for example automated loan application approvals, the trade-off between model accuracy and interpretability is critical. While deep learning models might offer better predictions, the need for transparency, fairness, and regulatory compliance often makes simpler, more interpretable models like decision trees. Articulating these trade-offs helps stakeholders understand the rationale behind model choices and ensures that the system aligns with legal and ethical standards.

Algorithm complexity impacts a model’s interpretability and generalization. Articulating these trade-offs is crucial for understanding model design and limitations. Reviewing the business logic of a machine learning model is important to avoid human errors. When we changed data analysts, the new analyst didn’t understand the model and included variables that skewed the results. For instance, before predicting stock levels, clustering inventory (ABC analysis) is essential. The new analyst overlooked this, leading to completely inaccurate stock predictions.

While model performance metrics like Accuracy, Recall, Precision, F1-score indicate that the model performs well technically. End-to-end business metrics like customer retention retention, Revenue impact (GMV growth) ensure that the solution drives meaningful business outcomes.

The engineer's responsibility is to translate the ML model's performance metrics into tangible business outcomes, aligning them with user experience and application efficiency. However, no software solution is perfect; every approach has trade-offs, including performance metrics. The choice of metric should be driven by business needs. Accuracy may suit uniformly distributed data, but for imbalanced real-world data, F1-score can be more appropriate as it better captures minority class performance.

Tell the stakeholder clearly what you have optimized the model on. A model optimized on F1 score may not work on a credit fraud problem. More importantly use non-tech jargons to explain this. You don't have to sound like Albert Einstein explaining these. Similarly talk about the thresholds used in case it's a classification problem and the reason for the choice. Also discuss some of the monitoring plans when it comes to tracking the model performance and the conditions you can see some of the metrics drift (sudden war, new regulation). More importantly tell them that AI models are never perfect despite all efforts but can be a force multiplier if used well.

It helps to explain in detail what the meeting mean and how to interpret them from business perspective Target than technical. This has helped me a lot in the past as clients also sometimes want to see all of their options and perhaps might want to try optimising on metrics other than what we would have thought of.

I recommend taking a consultative approach with the subject matter expert regarding the business problem we are trying to solve with a machine learning model. Then we will understand what metrics to use, and also whether the model is a viable solution to the problem. For instance, if the business problem is about predicting whether the patient has cancer, the model may be tuned to deliver a high recall score while the precision score may suffer. The model will deliver the required outcome because minimizing false negatives is important, and having more false positives is still acceptable. Some business problems may require more balanced precision and recall scores.

It's important to assess if the complex model is the right fit for a specific requirement. Once, I had to build a text summarization feature for large volumes of data. I used an open-source LLM, but its response was slow due to resource constraints on the hosting node. After analyzing the requirements, we realized we'd only need the model occasionally. Using OpenAI's API would cost much less than hosting the resource-hungry open-source LLM. While complex models offer benefits, remember they come with trade-offs. There may be cheaper, more effective solutions for the same requirement. Finding the right balance between model complexity and practical considerations is key for optimal performance and resource utilization.

Energy consumption is a critical factor in the use of AI and machine learning (ML) due to the substantial power required for training and deploying models. Training large models like GPT-3 can consume as much electricity as 130 US homes annually, highlighting the environmental impact. Efficient energy use is essential to mitigate this, as AI's energy demand is projected to increase significantly. Techniques like optimizing algorithms, using renewable energy, and improving hardware efficiency are vital to reduce the carbon footprint. Addressing energy consumption in AI and ML is crucial for sustainable technological advancement

Don't be afraid to ask for help, no one is an island of knowledge, whenever you're stuck and you don't know what to do, you can always get help online.