Unlike traditional (multi-class) learning approaches that assume label independence, multi-label learning approaches must deal with the existing label dependencies and relations. Many approaches try to model these dependencies in the process of learning and integrate them in the final predictive model, without making a clear difference between the learning process and the process of modelling the label dependencies. Also, the label relations incorporated in the learned model are not directly visible and cannot be (re)used in conjunction with other learning approaches. In this paper, we investigate the use of label hierarchies in multi-label classification, constructed in a data-driven manner. We first consider flat label sets and construct label hierarchies from the label sets that appear in the annotations of the training data by using a hierarchical clustering approach. The obtained hierarchies are then used in conjunction with hierarchical multi-label classification (HMC) approaches (two local model approaches for HMC, based on support vector machines (SVMs) and predictive clustering trees (PCTs), and two global model approaches, based on PCTs for HMC and ensembles thereof). The experimental results reveal that the use of the data-derived label hierarchy can significantly improve the performance of single predictive models in multi-label classification as compared to the use of a flat label set, while this is not the case for ensemble models.
COBISS.SI-ID: 29561127
Multi-label classification (MLC) tasks are encountered more and more frequently in machine learning applications. While MLC methods exist for the classical batch setting, only a few methods are available for the streaming setting. We propose a new methodology for MLC via multi-target regression in a streaming setting. Moreover, we develop a streaming multi-target regressor iSOUP-Tree that uses this approach. We experimentally compare two variants of the iSOUP-Tree method (building regression and model trees), as well as ensembles of iSOUP-Trees with state-of-the-art tree and ensemble methods for MLC on data streams. We evaluate these methods on a variety of measures of predictive performance (appropriate for the MLC task). The ensembles of iSOUP-Trees perform significantly better on some of these measures, especially the ones based on label ranking, and are not significantly worse than the competitors on any of the remaining measures. We identify the thresholding problem for the task of MLC on data streams as a key issue that needs to be addressed in order to obtain even better results in terms of predictive performance.
COBISS.SI-ID: 30119463
In the context of the project, we addressed the task of multi-target regression, where we generate global models that simultaneously predict multiple continuous variables. We used ensembles of generalized decision trees, called predictive clustering trees (PCTs), in particular bagging and random forests (RF) of PCTs and extremely randomized PCTs (extra PCTs). We added another dimension of randomization to these ensemble methods by learning individual base models that consider random subsets of target variables, while leaving the input space randomizations (in RF PCTs and extra PCTs) intact. Moreover, we proposed a new ensemble prediction aggregation function, where the final ensemble prediction for a given target is influenced only by those base models that considered it during learning. We performed an extensive experimental evaluation on a range of benchmark datasets, where the extended ensemble methods were compared to the original ensemble methods, individual multi-target regression trees, and ensembles of single-target regression trees in terms of predictive performance, running times and model sizes. The results showed that the proposed ensemble extension can yield better predictive performance, reduce learning time, or both, without a considerable change in model size. The newly proposed aggregation function gave best results when used with extremely randomized PCTs. Finally, we also included a comparison with three competing methods, namely random linear target combinations and two variants of random projections.
COBISS.SI-ID: 31606055
In the context of the project, we addressed the task of feature ranking for multi-target (MTR) regression. This task is receiving an increasing attention from the research community. However, performing feature ranking in the context of MTR had not been studied so far. We propose three feature ranking score for MTR, i.e., the Symbolic, Genie3 and Random Forest scores, that can be calculated in tree ensemble learning. These scores are then coupled with three types of ensemble methods: Bagging, Random Forest, and Extremely Randomized Trees. All of the ensemble methods use predictive clustering trees (PCTs) as base predictive models. In total, we consider eight different ranking methods (score-ensemble pairs) and extensively evaluated these methods on 26 benchmark MTR datasets. The results reveal that all of the methods produce relevant feature rankings and that the best performing method is the combination of the Genie3 ranking score used with Random Forest ensembles of PCTs.
COBISS.SI-ID: 30862887
In this work, we address the task of hierarchical multi-label classification (HMLC). HMLC is a variant of classification, where a single example may belong to multiple classes at the same time and the classes are organized in the form of a hierarchy. Many practically relevant problems can be presented as HMLC tasks, including the problems of predicting gene function, habitat modelling, annotation of images and videos, etc. We propose to extend the predictive clustering trees for HMLC – a generalization of decision trees for HMLC – toward learning option predictive clustering trees (OPCTs) for HMLC. OPCTs address the myopia of the standard tree induction by considering alternative splits in the internal nodes of the tree. An option tree can also be regarded as a condensed representation of an ensemble. We evaluate OPCTs on 12 benchmark HMLC datasets from various domains. With the least restrictive parameter values, OPCTs are comparable to the state-of-the-art ensemble methods of bagging and random forest of PCTs. Moreover, OPCTs statistically significantly outperform PCTs.
COBISS.SI-ID: 30862631