Machine Learning (ML) has long established itself in our everyday lives, its applications are now everywhere [

18]. However, along with ML, there is also widespread awareness that, given a specific problem, the process of designing and implementing a truly effective and efficient ML system requires considerable skills [

19]. This process is also time-consuming and error-prone [

17]. It is, hence, natural that the academic and industrial worlds have turned their attention to the idea of automating this process as much as possible [

20]. Therefore, studies documenting such research efforts have been reported in the literature since the 1990s [

21]. This interest has become even more consolidated in recent years with the emergence of Deep Learning, as evidenced by the publication of excellent surveys and reviews on AutoML. Among these, Hugo J. Escalante [

22] provides an introduction to AutoML by referring to the overview proposed in [

23]. He also presents a historical review in chronological order of the main contributions put forward in the last decade in the context of AutoML for supervised learning. In [

24], the authors review the AutoML approaches based on the complete pipeline including data preparation, feature engineering, model generation, and, finally, model evaluation. In particular, they focus on Neural Architecture Search (NAS) algorithms, also providing an experimental evaluation and comparative analysis on the ImageNet (

http://www.image-net.org/ (Accessed: 6 January 2021)) and CIFAR-10 (

https://www.cs.toronto.edu/~kriz/cifar.html (Accessed: 6 January 2021)) datasets. NAS can be seen to all intents and purposes as a subfield of AutoML [

20] and presents also significant common traits with Meta Learning [

25] and hyperparameter optimization [

26]. Elshawi et al. [

27] focus on the models and methods proposed to achieve the partial or total automation of the integrated Combined Algorithm Selection and Hyperparameter optimization (CASH process), as formally defined in [

28]. This process aims to reduce the human expert role as much as possible, allowing even non-expert users to develop their ML systems able to best meet their specific needs. As in [

24], also in [

27] the focus is on automatic NAS. Refs. [

17,

29,

30,

31] focus on the analysis of NAS algorithms as well. In particular, Elsken et al. [

17] provide a comprehensive review of the State of the Art in the NAS domain, classifying existing systems according to the basic components of a typical NAS process: search space, search strategy, and evaluation strategy. Wistuba et al. [

30] furnish a detailed analysis comparing the different existing AutoML approaches. They also present an in-depth discussion of architecture search spaces and architecture optimization algorithms based on the principles of Reinforcement Learning and evolutionary algorithms. In [

31], the authors illustrate the techniques proposed in the literature for automated feature engineering, automated model and hyperparameter learning, and automated deep learning. More specifically, they analyze the approaches based on gradient, Bayesian Optimization [

32], evolutionary algorithms, and Reinforcement Learning. They also review the most popular AutoML tools. Ren et al. [

29] analyze in-depth what has been done so far in the NAS. However, they follow a different perspective than in [

17,

30]. First, they explore the earlier NAS algorithms proposed in the literature, highlighting their characteristics and criticalities. Then, they provide the solutions adopted by subsequent NAS methods. Differently from previous works, in [

9,

33] the focus is not on NAS. In Zoller et al. [

33], the authors formulate the problem of the creation of the AutoML pipeline as a problem of mathematical minimization and present the approaches proposed in the literature to solve each step of the pipeline. In particular, they focus on classical ML approaches rather than neural networks. Finally, they provide an experimental evaluation of recent AutoML approaches (especially, hyperparameter optimization algorithms) and open-source AutoML frameworks on synthetic and real data. This comparative analysis is further extended by the same authors in [

21], thus providing readers with the most comprehensive experimental evaluation of AutoML tools reported in the literature. Refs. [

9,

20] also provide excellent overviews of the AutoML domain, but compared to [

21,

33] they cover fewer steps of the AutoML pipeline and do not report experimental tests performed on the documented approaches. In particular, in [

9], the authors—besides providing an exhaustive critical analysis of the State of the Art—present a general AutoML framework, which can be useful in the classification of the AutoML approaches proposed in the literature as well as in the design of new models. In [

34], the authors review the most relevant AutoML approaches proposed in the literature, verifying their possible practical application in a business context also through an experimental analysis conducted on independent benchmarks. As in [

27], also Tuggener et al. deeply analyze the systems that address the CASH problem. Waring et al. [

10] follow the same analysis scheme as proposed in other surveys (e.g., in [

9]), but focus on the healthcare field, one of the scenarios most interested in the automated design of ML and search algorithms. Therefore, they present the existing AutoML applications in this area and indicate the classic criticalities and opportunities in a healthcare setting.

Our work is in no way intended to be an alternative to the excellent works mentioned above. We explore the strengths and weaknesses of the ML models proposed in the literature to put forward their use—alone or in combination with other approaches—to provide possible valid AutoML solutions. We critically analyze those solutions from a theoretical point of view and we evaluate them empirically on a classic test domain, namely, that of the Atari games from the Arcade Learning Environment. Our goal is to identify what—we postulate—could be promising ways to create truly effective AutoML frameworks, capable of replacing the human expert as much as possible, thus facilitating the process of applying ML approaches to classic problems of specific domains.