Step Away From “The State Of The Art”
What is this blog about? It’s about taking a step back to move forward more effectively. Striving to be close to the state of the art is a challenge that most companies (including developers) should not be pursuing. The farther we are from “the state of the art”, the better.
Historical Context
For many years, the field of Artificial Intelligence has been quite undervalued. While societal perceptions of its value have changed, they have done so in a discriminatory manner. As the spotlight focuses on Generative AI, other subfields of AI have been sidelined. It’s notable how we refer to everything that isn’t Generative AI as 'Classic Analytics' with a somewhat dismissive tone.
Generative AI is just one tool among many in a toolbox, and we need to learn which tool is best suited for which scenarios. Companies and individuals are obsessed with trying to apply Generative AI, and there is a tendency to constantly chase the "state of the art" through papers and publications. But can this be counterproductive? The reality is that it can be.
What is considered "state of the art" today can become obsolete in the blink of an eye. Additionally, many papers focus on incremental improvements that may not have a significant impact on real-world applications. It’s also worth noting that this trend is appealing to many researchers, leading to a proliferation of papers, few of which have lasting influence.
What Should Be Done
The primary focus for most companies and professionals should be to establish a solid foundation in data engineering. Secondly, start addressing problems with other AI tools that have already reached a considerable level of maturity. And let me clarify, it’s fine to explore “novel” technologies and try to stay informed, but a balance between innovation and pragmatism is required.
"Classic Analytics" offers a vast field of business opportunities and a multitude of challenges to solve and optimize. It has the potential to deliver tangible and significant results for the business. You can achieve substantial financial success and innovate with Machine Learning; in fact, you are likely to be more successful because the community has more background and experience in taking projects from experimentation to production.
In my opinion, one of the several reasons why many companies are stagnating in the process of adoption or conversion into a data-driven organization is distraction and a lack of pragmatism. Without a solid foundation in engineering, there is no AI. This applies to both companies and developers.
Executing ML projects is not easy. Next, we will analyze the complexity of an ML project with a productive approach (from a high level) and the knowledge and tasks involved in moving these projects from experimentation to production.
What a Machine Learning Project Involves
Let's set aside business knowledge and focus, practically, on analyzing part of the technical foundation required to carry out such a project with a productive approach. We’ll avoid discussing specific terms and take a more "generalist" perspective:
1) Data Engineering: We need to ingest information, analyze it, interpret it, partition the data (training, validation, testing), clean the data, transform it (handle missing data, outliers, normalize data, encoding, etc.), and save this data again (in multiple formats/structures), etc.
1Downloading a Jupyter Notebook with test code and data might seem easy, but in practice, it’s not that simple to go from toy project to production.
2) Knowledge of Modern Architectures, including:
Container orchestration (Docker, Kubernetes).
Microservices architecture.
Experience in Cloud & FinOps (e.g., if you deploy on AWS, you’ll likely need to know about EventBridge, Lambda, SageMaker, Databricks, S3, managing access in AWS IAM, CloudWatch, etc.)
Distributed systems and processing. Scalability and performance optimization (caching, in-memory storage, etc.)
3) DevOps/MLOps Practices: This area involves several domains, but to be pragmatic... A production grade model cannot be conceived without CI/CD. It will certainly require the deployment of new releases and process automation (e.g., testing when code changes are made). It also requires a structure to evaluate the generated model against what is in production as a solution (Shadow Testing) or between different models (A/B testing).
4) Model Tracking:
When starting an ML project, the first thing we need to define is a baseline to start from and what we aim to improve. This baseline, along with other defined indicators, is crucial for setting our success goals and measuring our progress. Without specific model monitoring data, we won’t be able to detect biases, model drift, evaluate performance, or identify other issues affecting our service.
There are tools on the market, such as Comet, that allow us to manage, monitor, and especially provide visibility into insights generated automatically (compatible with Databricks, of course).
5) Security and Privacy:
From a regulatory perspective, it is imperative to comply with data protection laws. Rigorous controls must be implemented for data access and secure management of elements such as encryption keys.
Sensitive information masking involves anonymization and pseudonymization to protect individuals' identities and prevent unauthorized disclosure of confidential information. However, in practice, this concept falls under the domain of data engineering.
6) Service Exposure:
Whatever it is, the way you tell your story online can make all the difference.
This means that knowledge of APIs and web services (e.g., REST), message and queue services, events, integration with business systems in the ecosystem, etc., is also required. It’s true that some of these processes are partially automated or self-managed by certain platforms.
Solid Foundation in Algorithms: Understanding algorithms goes beyond using (which is not the same as applying) regression algorithms, decision trees, or neural networks. It’s also not just about using ML frameworks like TensorFlow, PyTorch, or scikit-learn.
The reality is that importing these libraries or reusing code is very easy; the challenge lies in understanding the mathematical and statistical foundations, knowing when and how to apply them in practice (especially when training models from scratch, fine-tuning, or progressing to more complex techniques such as optimization or hyperparameter adjustments, or for model interpretability and explainability).
I acknowledge that many other domains are not covered here, or have been grouped (MLOps and Evaluation), but the goal here is not to delve into detail or step-by-step construction of a Machine Learning model (we can cover that in another blog), but to highlight the knowledge and experience required by teams. These are achievable projects but with a high degree of complexity.
On the other hand, it’s worth mentioning that when approaching these projects, starting with a discovery phase is essential (especially if we are a consultancy developing a custom service) and the working methodology should be agile. It is neither feasible nor advisable to approach these projects in any other way.
Opportunity and Final Conclusions
The best thing that Generative AI has done is encourage clients to use these tools and address problems that, despite being solvable for a long time with "Classic Analytics," were overlooked or simply not considered until the recent wave of hype. In many cases, when people think of AI, they think of GPT, but in practice and from a commercial and business perspective, it doesn’t matter. This has helped spark curiosity and will encourage clients to modernize. I am sure this will continue to generate more data engineering, machine learning, and of course, Generative AI projects as well.
Society has opened its eyes and is willing to invest in these types of projects. Respect for these technologies and/or practices has grown (which, as we mentioned at the start of the blog, was not always the case). Back in my college days, I remember I once proposed my university thesis to be solved with neural networks and hearing some laughter or being called crazy. Today, society listens and is interested in these topics.
However, I am convinced that there is still work to be done on the culture side needed to position these practices where they should be, and I am not just talking about ML/AI as adopted terminology. I am especially referring to the foundation of all this, which is the mathematics and physics that govern everything.