Run-Time Hierarchical Management of Mapping, Per-Cluster DVFS and Per-Core DPM for Energy Optimization

Abstract

Heterogeneous cluster-based multi/many-core systems (e.g., ARM big.LITTLE, supporting dynamic voltage and frequency scaling (DVFS) at cluster level and dynamic power management (DPM) at core level) have attracted much attention to optimize energy on modern embedded systems. For concurrently executing applications on such a platform, this paper aims to study how to appropriately apply the three system configurations (mapping, DVFS, and DPM) to reduce both dynamic and static energy. To this end, this paper first formulates the dependence of the three system configurations on heterogeneous cluster-based systems as a 0–1 integrated linear programming (ILP) model, taking into account run-time configuration overheads (e.g., costs of DPM mode switching and task migration). Then, with the 0–1 ILP model, different run-time strategies (e.g., considering the three configurations in fully separate, partially separate, and holistic manners) are compared based on a hierarchical management structure and design-time prepared data. Experimental case studies offer insights into the effectiveness of different management strategies on different platform sizes (e.g., #cluster × #core, 2 × 4, 2 × 8, 4 × 4, 4 × 8), in terms of application migration, energy efficiency, resource efficiency, and complexity.

1. Introduction

Heterogeneous cluster-based multi/many-core platforms (e.g., ARM’s big.LITTLE architectures [1,2]) have attracted much attention as a solution to deliver both high-performance and energy-efficient embedded systems. Such a platform groups homogeneous cores into clusters, and exploits heterogeneity by using different core types among clusters. In addition, the platform allows updating voltage/frequency ($v/f$) levels of each cluster via dynamic voltage and frequency scaling (per-cluster DVFS, for dynamic power/energy optimization) and switching off each idle core via dynamic power management (per-core DPM, for static power/energy optimization). The diversity of core types, cluster $v/f$ configurations and the possibility of shutting off idle cores provides many opportunities for energy optimization of the overall system.

Heterogeneous cluster-based systems allow multiple applications to execute concurrently. A set of simultaneously executing applications can be defined as a use case according to [3]. From one use case to another, the change in performance demands lead to the need for run-time mapping. Run-time mapping has been often combined with DVFS and DPM to achieve power/energy efficiency in state-of-the-arts (SotA). The SotA strategies are compared in

Table 1. Comparison of applying mapping, DVFS and DPM in SotA.

Management Strategy		References	Applications
❶	Mapping → DVFS	[4,5,6,7]	Multiple concurrent apps
	Mapping & DVFS	[8,9,10]	One single app
		[11,12,13]	Multiple concurrent apps
❷	Mapping → DVFS* & DPM	[14,15]	Periodic independent tasks
	Mapping & DVFS* → DPM	[16]	one single app
❸	Fully-separate: Mapping → DVFS → DPM	This work	Multiple concurrent apps
	Partially-separate: Mapping & DVFS → DPM
	Holistic: Mapping & DVFS & DPM

Note: → refers to separate manner. & refers to holistic manner. app: an application having dependent tasks. DVFS: per-cluster DVFS by default. DVFS*: per-core DVFS.

. As shown in ❶ rows, there have been two approaches to combining mapping and DVFS in the literature: (1) separately (denoted by → ) and (2) holistically (or integratedly, denoted by & ). The first approaches (i.e., mapping → DVFS), such as [4,5,6,7], separate application mapping and DVFS into two subsequent steps. After application mappings are fixed, $v/f$ configurations are set according to the system-performance constraints. The second approach (i.e., mapping & DVFS) is often used in recent works [8,9,10,11,12,13]. Such approaches apply task mapping and DVFS in a holistic manner. These approaches take into account the mutual influence between mapping and DVFS, and any changes in the mapping can trigger an update of $v/f$ configurations. The holistic approaches can achieve a more energy-efficient solution, and may come at the cost of exploring complexity [6]. Due to the exploring complexity, the recent works of [9,10] consider the mapping and DVFS configuration for one multi-task application. For concurrently executing applications, some recent works [10,11,13] reduce exploration complexity by using heuristic strategies.

DMP has been combined with mapping and DVFS in multi-core systems such as in [14,15,16] (see ❷ rows in Table 1). These works consider independent tasks or one single application (can have multi-tasks) mapping on a per-core DVFS platform (denoted by DVFS* in Table 1). These approaches are limited for our considered case where multi-task applications are concurrently executing on a per-cluster DVFS platform. On the other hand, DPM is rarely used in ARM big.LITTLE systems in existing works due to two main reasons: (1) extra DPM overheads of switching-on/-off cores can be much higher than DVFS overheads [17]; (2) the limited resources of the typical ARM big.LITTLE architecture (i.e., ODROID-XU4 Exynos 5 Octa (5422) [1], 4 cores in each of the LITTLE and big clusters) make idle cores or a whole idle cluster a rare situation for multiple concurrently executing applications. However, for a future platform with more cores/clusters (e.g., ODROID-MC1 [2], an extension of ODROID-XU4 in a single board consisting of 4 LITTLE clusters and 4 big clusters), shutting off idle cores/clusters can have energy benefits. Moreover, driven by the continued success of reducing the technology size of transistors in the semiconductor industry, leakage power has increased its dominance in total power consumption [18]. It is significant to trade off static and dynamic energy optimization.

The focus of this paper is to study how to appropriately apply DPM with mapping and DVFS to have an overall energy optimization (i.e., including dynamic and static parts) for multiple applications concurrently executing on a heterogeneous cluster-based system. This paper presents three possible management strategies, and highlights their differences (see ❸ rows in Table 1). The first two management strategies include mapping → DVFS → DPM and mapping & DVFS → DPM), which can be respectively regarded as fully separate and partially separate strategies. The two strategies perform mapping and DVFS according to existing strategies (e.g., [4,11]; see ❶), and then switch off cores that are not assigned workloads to reduce static energy. These two straightforward extension strategies can achieve lower cluster $v/f$ configurations (DVFS) by using more cores (depending on mapping) in each cluster for dynamic energy optimization. However, because there are fewer idle cores that can be shut off (DPM), static energy can be increased. Due to such dependencies of the three system configurations, a holistic strategy (i.e., mapping & DVFS & DPM) can be a promising solution for better dynamic and static energy tradeoffs. The proposed holistic management strategy (for mapping, per-cluster DVFS and DPM) is the key novelty of our work. Most of the existing works focus on holistic strategies without DPM consideration (see ❶) or they consider per-core DVFS (see ❷). Our study shows that the partially separate and holistic strategies outperform each other on different platform sizes (in term of #clusters or #cores, see Section 4). To the best of our knowledge, such insights are not seen in existing approaches.

In large-scale systems (e.g., with more clusters/cores), the complexity and scalability of management strategies are challenging. As summarized in this survey [19], run-time strategy complexity can be handled by hybrid management, which reduces run-time computation complexity based on some design-time prepared data. The scalabilty issue is usually resolved by hierarchical management structure [11,20], which typically uses two-level hierarchical management, consisting of a global and a local management strategy. Based on the hybrid hierarchical management structure (HHMS) in [11], this paper studies the effectiveness of different management strategies (e.g., partially separate, holistic) on heterogeneous cluster-based platforms. The demonstration of the HHMS [11] adaptability (e.g., to DPM consideration and different management strategies) is another novelty of this paper. In summary, the main contributions of this paper are as follows.

• A 0–1 ILP model is presented to establish the dependencies of mapping, per-cluster DVFS and per-core DPM. The objective is to optimize dynamic and static energy of different sets of concurrently active applications (in varied use cases) on heterogeneous cluster-based platforms, taking into account possible run-time configurations overheads due to application migration and DPM mode switching.
• With the 0–1 ILP model, different strategies (i.e., fully separate, partially separate and holistic) are realized into global and local management of the HHMS [11], based on some design-time prepared data. The differences and benefits of the management strategies are illustrated.
• Experimental evaluations are performed for 1023 use cases (i.e., generated in random orders for 10 applications) and on different platform sizes (#cluser × #cores, 2 × 4, 2 × 8, 4 × 4, 4 × 8). The experiments offer insights into the effectiveness of the different management strategies (fully separate, partially separate, and holistic) in terms of application migration, energy efficiency, resource efficiency, and complexity.

The rest of this paper is organized as follows: Section 2 presents the backgrounds of HHMS [11] and presents the motivation for this paper. Section 3 formulates the 0–1 ILP model and presents the studied HHMS strategies (fully separate, partially separate, and holistic). Section 4 demonstrates and discusses the experimental results. Finally, Section 5 concludes this paper.

2. Background and Motivation

This section presents the background on HHMS [11] and illustrates the motivation of this paper. For reference,

Table 2. Notations used in this paper.

System configuration
$HHMS$	Hybrid hierarchical management structure
$DVFS$	Dynamic voltage and frequency scaling
$DPM$	Dynamic power management
Application level, platform level, and management level
$ap{p}_i$	An application with index i
$Perio{d}_{ap{p}_i}$	The period of an application $ap{p}_i$
$u_m$={$ap{p}_1,ap{p}_2,...,ap{p}_I$}	A use case $u_m$ consists of a set of simultaneously active applications
I	The number of activate applications in a use case
$cluste{r}_j$	A cluster with index j
J	The number of available clusters in the platform
$N_j$	The number of available cores in $cluste{r}_j$
$\{f_{j,1},f_{j,2},...,f_{j,max}\}$	The available discrete frequency levels of $cluste{r}_j$
$GM$	Global management
$LM$	Local management
$MA{F}_{i,j}^c$	The minimum allowed frequency (MAF) of $ap{p}_i$ mapped on c cores in $cluste{r}_j$
$c^{max}$	The maximum number of cores used by design-time-prepared mapping for an application
$MA{F}_j^{LC}$	The lowest common MAF of the mapped applications in $cluste{r}_j$
$X_{ap{p}_i}^c$	The design-time mapping of $ap{p}_i$ mapped on c number of cores
$N_j^{used}$	The number of available cores in $cluste{r}_j$
$N_j^{used}$	The number of used cores in $cluste{r}_j$
$M{a}_{i,j}$	The run-time application-to-cluster mapping of $ap{p}_i$ to $cluste{r}_j$
$f_j$	The run-time configured frequency of $cluste{r}_j$
Energy model
$E_{sys}$	The overall energy of system
$E_{sys}^{dynamic}$	The dynamic energy of system
$E_{sys}^{static}$	The static energy of system
$E_{overhead}^{DPM}$	The overhead energy of DPM mode switching
$E_{overhead}^{migration}$	The overhead energy of application migration
Experiment
FS	Fully separate strategy, mapping → DVFS → DPM
PS	Partially separate strategy, mapping & DVFS → DPM
H	Holistic strategy, mapping & DVFS & DPM
$Mx$	Allowing x migration(s) across clusters
$SER$	Static energy ratio, the ratio of static energy compared to dynamic energy
$\#LCM$	Least comment multiple of the active application periods in each use case
$\#Migration$	least comment multiple of the active application periods in each use case
$FCFS$	First come first served task mapping strategy in local management
$N_{all}^{used}$	The number of used cores of all clusters

lists the parameters and other variables used in this paper.

2.1. System Models and HHMS

Figure 1 depicts the HHMS in [11]. The management level of the system is linked to the application level and platform level. At the application level (see part a), periodic multi-task applications are considered. The period of an application $ap{p}_i$ is denoted as $Perio{d}_{ap{p}_i}$. Each application consists of a set of computation tasks and a set of communication edges representing dependencies among the tasks. A set of simultaneously active applications is defined as a use case [3]. For instance, $u_1=\{ap{p}_1,ap{p}_2\}$ refers to the use case consisting of two active applications.

At the hardware level (see part b), the platform is composed of J clusters. Each cluster, indexed by $cluste{r}_j$, is composed of an $N_j$ number of cores. The core type within each cluster is the same, and core types between clusters can be different. The communication resources among clusters and within each cluster can be either a bus or a network-on-chip (NoC). In addition, the considered platform supports per-cluster DVFS. All the cores within a cluster share the same $v/f$ level. The available discrete frequency levels of $cluste{r}_j$ are denoted as {$f_{j,1},f_{j,2},\dots ,f_{j,max}$} and operating voltages are adapted to the frequency. The platform also supports per-core DPM, which can switch off any idle core to reduce leakage power/energy.

At the management level (see part c), the HHMS [11] performs hybrid management, where design-time data is prepared to reduce run-time computation burdens and still guarantee mapping quality [21]. For each application, the design time-prepared data includes the following.

• A set of mappings using a different number of cores (i.e., different threads). $X_{ap{p}_i}^c$ denotes the prepared mapping of $ap{p}_i$ mapped on c number of cores.
• A set of minimum allowed frequencies (MAFs). MAF refers to the minimum frequency that allows the application to respect its timing constraint. Each prepared mapping has its own MAF. $MA{F}_{i,j}^c$ denotes the MAF of $X_{ap{p}_i}^c$ in $cluste{r}_j$.

For example, Figure 1 (c.1) presents the MAFs prepared for $ap{p}_1$ design-time mappings, which use one core ($X_{ap{p}_1}^1$) and two cores ($X_{ap{p}_1}^2$). $MA{F}_{1,j}^2$ is smaller than $MA{F}_{1,j}^1$ because the higher performance of more threads ($X_{ap{p}_1}^2$) allows execution at a lower frequency level under the timing constraint ($Perio{d}_{ap{p}_i}$). In particular, $MA{F}_{i,j}^{c_{max}}$ is used to denote the MAF of the design-time mapping by using the maximum number of cores for $ap{p}_i$. It means that $MA{F}_{i,j}^{c_{max}}$ is the minimum of all prepared MAFs for an application (e.g., $MA{F}_{1,j}^{c_2}$ for $ap{p}_1$ in part (c.1), $MA{F}_{2,j}^{c_3}$ for $ap{p}_2$ in part (c.2) of Figure 1).

Based on the design-time-prepared mappings and MAFs, the HHMS [11] performs run-time management via two hierarchical levels for scalability consideration. For each use case, the global management (GM) determines the application-to-cluster mapping, wherein each application is assigned to one cluster to avoid costly communications among clusters. For the assigned applications in each cluster, the local management (LM) determines the task-to-core mapping. A run-time management example is shown in Figure 1 (c.3), where $ap{p}_1$ and $ap{p}_2$ (e.g., in $u_1$) are mapped to the same cluster ($cluste{r}_j$) and the design-time mappings are selected according to MAFs. Due to per-cluster DVFS, all applications mapped in $cluste{r}_j$ should execute at the same frequency level. Let $f_j$ denotes the run-time configured frequency of $cluste{r}_j$. In [11], the selected mapping corresponds to the ones supporting the lowest common frequency (denoted by $MA{F}_j^{LC}$), which is equal to the highest $MA{F}_{i,j}^{c_{max}}$ ($MA{F}_{1,j}^2$ in this example) of all assigned applications in $cluste{r}_j$. Then the selected mappings (e.g., $X_{ap{p}_1}^2$ and $X_{ap{p}_2}^2$, as indicated by the red dotted line in Figure 1 (c.3)) can be combined by a task-to-core mapping strategy at $f_j=MA{F}_j^{LC}$.

The HHMS in [11] simplifies the mapping problem by linking task-to-core mapping to the cluster frequency configuration (per-cluster DVFS). This is the reason why the HHMS in [11] is selected as the baseline management structure in this paper. From here on, mapping refers to application-to-cluster mapping by default if not specified. Based on the HHMS, different GM and LM strategies can be used according to the optimization objectives. In this paper, LM uses the straightforward first come first served (FCFS) strategy [11], whereby the design-time prepared mapping is directly used at run time. One core is allowed to be only mapped with tasks of one application. The number of used cores in a cluster is the sum of cores used by the selected mappings. FCFS is selected due to its simplicity and feasibility in both virtual prototype simulation (e.g., Gem5) [7] and real operating system [4]. More details about the system model and HHMS can refer to [11].

2.2. Motivation of This Paper

This paper aims to study the effectiveness of different management strategies (applying mapping, DVFS, and DPM in fully separate, partially separate, and holistic manners) on heterogeneous cluster-based platforms based on the HHMS [11]. To better illustrate the motivation, this section first introduces the limitations of [11] (see benefits in Section 2.1), and then discusses possible energy improvement by using HHMS. Figure 2 shows energy optimization possibilities of $u_1=\{ap{p}_1,ap{p}_2\}$. For clarity, let $M{a}_{i,j}$ denote the run-time application-to-cluster mapping of $ap{p}_i$ in $cluste{r}_j$. For the two applications in a two-cluster platform, there are four (i.e., $J^I$ in general) application-to-cluster mapping possibilities (❹ column of Figure 2). The first two mappings (in case 1, $\{M{a}_{1,1},M{a}_{2,1}\}$ and $\{M{a}_{1,2},M{a}_{2,2}\}$) map the two applications to the same cluster. The third and fourth mappings (in case 2, $\{M{a}_{1,1},M{a}_{2,2}\}$ and $\{M{a}_{1,2},M{a}_{2,1}\}$) divide two applications into two different clusters.

Different application-to-cluster mappings lead to different cluster frequency $f_j$ configurations as well as task-to-core mappings (due to different design-time $X_{ap{p}_i}^c$ selections). As previously explained in Section 2.1, the work of [11] sets $f_j$ according to the lowest common MAF (i.e., $MA{F}_j^{LC}$, highlighted in red in the column ❺) of the mapped applications in the same cluster. Once $f_j$ is selected for each cluster, the selection of design-time $X_{ap{p}_i}^c$ can be known based on MAFs [11]. In the example of $\{M{a}_{1,1},M{a}_{2,1}\}$, $f_j=MA{F}_{2,1}^2$ indicates the task-mapping selections $X_{ap{p}_1}^2$ and $X_{ap{p}_2}^2$ (see red dotted line in part b of Figure 2). Thus the number of used cores in $cluste{r}_1$ is 4 ($N_1^{used}=4$, because each selected design-time mapping uses two cores/threads). On the other hand, when each cluster has a limited number of available cores (e.g., $N_j=3$), the selected design-time mappings based on $MA{F}_j^{LC}$ [11] can be infeasible (e.g., in case.1, due to $N_j^{used}(X_{ap{p}_1}^2,X_{ap{p}_2}^2)>N_j$). The strategy of [11] discards the infeasible task-to-core mappings (at $f_j=MA{F}_j^{LC}$) in case 1, and it continues to explore energy-efficient configurations (mapping and $f_j$) only in case 2.

If task-mapping exploration at $f_j>MA{F}_j^{LC}$, the overall energy consumption can be further optimized under resource constraints (see column ❻). In case 1, when $f_j$ shifts from $MA{F}_{1,j}^2$ (in red) to $MA{F}_{2,j}^1$ (in blue), the corresponding selected mappings shift to $X_{ap{p}_1}^2$ and $X_{ap{p}_2}^1$ accordingly. $N_j^{used}$ can be reduced to 3, satisfying resource constraint. In this way, the cluster frequencies are minimized (in both case 1 and case 2) for dynamic energy (as $Powe{r}_{dynamic}\propto$ $V_{dd}^2\times f_j$ [22], $V_{dd}$ is the supply voltage). Moreover, allowing task-mapping exploration at $f_j>MA{F}_j^{LC}$ can enable tradeoffs between dynamic energy and static energy (see column ❼). In case.1, if $f_j$ increases from $MA{F}_{2,j}^1$ (in blue) to $MA{F}_{1,j}^1$ (in purple), $N_j^{used}$ can decrease from 3 to 2. This allows switching off one core (DPM) to save static energy. Due to different static and dynamic energy tradeoffs, lower energy of the overall system can be achieved.

However, for each application-to-cluster mapping (in case 1 and case 2), there can be many possibilities of cluster frequency (or MAF) configurations and core shutting-off decisions. The dependencies of application-to-cluster mapping per-cluster DVFS and DPM (observed from ❹, ❻, and ❼) motivate this paper to develop a management strategy (which can be in fully separate, partially separate, and holistic manners). The optimization metrics include low energy consumption of the overall system and low strategy complexity. Additionally, the developed strategies should take into account run-time configuration overheads, such as DPM mode-switching overheads and application-migration overheads (e.g., due to different application-to-cluster mappings from one use case to another). Notice that the run-time configuration overheads are often neglected in existing woks [8,9,10,11,12,13].

3. 0–1 ILP Model Formulation and Proposed Solution

In this section, the energy optimization problem is formulated into a 0–1 ILP model. Based on the problem formulation and HHMS [11], different management strategies (fully separate, partially separate, and holistic) are presented.

3.1. 0–1 ILP Problem Formulation

Given: A set of periodic multitask applications (in a use case) to be executed concurrently on a heterogeneous cluster-based platform supporting per-cluster DVFS and per-core DPM.

Find: Application-to-cluster mapping, task-to-core mapping, cluster frequencies and switching-off cores, when use case varies (e.g., an application finishes or joins).

• Application-to-cluster mapping: The mapping of active applications (in each use case) on a cluster-based platform is defined by a matrix variables${\left[a_{i,j}\right]}_{I\times J}$, where $a_{i,j}$ is a binary variable equal to:

  $$a_{i,j}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}ap{p}_i\mathrm{is}\mathrm{assigned}\mathrm{onto}cluste{r}_j.\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

  (1)
• Cluster frequency configurations (DVFS): The optimized $f_j$ is set according to the all MAFs of the mapped applications in a cluster. $f_j$ should not be smaller than the lowest common MAF ($MA{F}_j^{LC}$), which refers to the maximum $MA{F}_{i,j}^{c_{max}}$ value of all assigned applications in $cluste{r}_j$ (Equation (2)). In addition, $f_j$ configuration should be within cluster frequency range ( Equation (3)).

  $$f_j>=MA{F}_j^{LC}=max\{MA{F}_{i,j}^{c_{max}}\times a_{i,j}\},\forall i$$

  (2)

  $$f_j\in \{f_{j,1},f_{j,2},\dots ,f_{j,max}\},\forall j.$$

  (3)
• Task-to-core mapping: The mapping of tasks to cores should respect resource constraint in each cluster. $N_j^{used}$ should not be larger than the available cores ($N_j$) in a cluster. $N_j^{used}$ depends on the application-to-cluster mapping matrix variables, and on the applied local management strategy. This paper uses the FCFS [11] task-to-core mapping strategy and the corresponding $N_j^{used}$ is the sum of the used cores of selected $X_{ap{p}_i}^c$ (corresponding to $f_j$) in each cluster.

  $$N_j>=N_j^{used}({\left[a_{i,j}\right]}_{I\times J},f_j,X_{ap{p}_i}^c).$$

  (4)
• Switching-off cores (DPM): The idle cores without any mapped tasks can be switched off. Note that Equations (4) and (5) highlight the dependencies between mapping (of both application level and task level), frequency configuration, and DPM.

  $$N_{off}=N_j-N_j^{used}$$

  (5)

Objective: Minimizing the energy consumption of the overall system. The system energy is the sum of dynamic energy ($E_{sys}^{dynamic}$), static energy ($E_{sys}^{static}$), and system run-time configuration overheads ($E_{overhead}^{DPM}+E_{overhead}^{migration}$). In our assumptions, the DVFS overhead is negligible compared to $E_{overhead}^{DPM}$ [17]. Moreover, $E_{overhead}^{migration}$ refers to the cost of migrating an application from one cluster to another, whereas the cost of migrating tasks at the core level is assumed to be negligible.

$$min{E}_{sys}=E_{sys}^{dynamic}+E_{sys}^{static}+E_{overhead}^{DPM}+E_{overhead}^{migration}$$

(6)

Subject to: Meeting platform resource constraints and application performance constraint of each application. These constraints have been expressed as Equations (2)–(4).

3.2. Proposed Management Strategies

Solving the above 0–1 ILP optimization problem is computationally expensive. Because obtaining the optimal task mapping on a multi-/many-core system is an NP-hard problem [3], combining mapping with DVFS and DPM makes exploration space even larger. In order to achieve optimized 0–1 ILP solutions at a reasonable time, this paper uses heuristic strategies based on HHMS [11]. Inspired by the existing works (as listed in Table 1), three heuristic management strategies are proposed to apply application mapping, DVFS and DPM in fully separate, partially separate, and holistic manners, denoted as Mapping → DVFS → DPM, Mapping & DVFS → DPM and Mapping & DVFS & DPM, respectively.

Figure 3 depicts the flowcharts of the three management strategies which are implemented in HHMS [11]. These three management strategies take as input the active application (in a use case detected at run time), the platform model, and design-time prepared data (Section 2.1) and provide an optimized mapping (application-to-cluster and task-to-core), per-cluster DVFS and per-core DPM configuration as output. In the run time-detected use case, the concurrently active applications are considered one after another (e.g., in random order). Based on HHMS and the formulated 0–1 ILP model, the global management (GM) determines application-to-cluster mapping (Equation (1)). After one application is mapped to a cluster, the next application is considered. The local management (LM) in each cluster determines task-to-core mapping, cluster frequency configurations (with regard to application-performance constraints and platform-resource constraints, Equations (2)–(4)), and core switching-off decisions (Equation (5)).

3.2.1. Fully Separate: Mapping → DVFS → DPM

Figure 3a illustrates the fully separate management strategy, which sequentially fixes the three system configurations. The GM uses the lowest power first (LPF) strategy in [4,23] for application-to-cluster mapping. LPF is a heuristic strategy which is often used for ARM big.LITTLE architectures [1]. For each active application, LPF first tries to map it to the lowest power cluster (e.g., LITTLE cluster). When the lowest power cluster cannot satisfy system constraints (application performance constraints, or platform resource constraints), LPF then maps the application to a higher power cluster (e.g., big cluster). This means that LPF provides clusters with different priorities for application-to-cluster mapping. Once the GM fixes every application mapping by LPF, the LM minimizes the cluster frequencies ($f_j$) subsequently to optimize $E_{sys}^{dynamic}$ (which can include $E_{overhead}^{migration}$). Finally, the LM switches off the idle cores in each cluster to reduce $E_{sys}^{static}$ (which can include $E_{overhead}^{DPM}$). In the fully separate strategy, the three system configurations (application-to-cluster mapping, DVFS, and DPM) are divided into three subsequent steps (Mapping → DVFS → DPM). The former configuration is independent of the latter.

3.2.2. Partially Separate: Mapping & DVFS → DPM

Figure 3b describes the partially separate management strategy. The strategy is a variant of [11], which considers the mutual impact between application mapping and cluster frequency ($f_j$) configurations. The GM maps each application to the cluster leading to the lowest $E_{sys}^{dynamic}$ (which can include $E_{overhead}^{migration}$ in Equation (6)) via a greedy strategy. For each active application, the GM attempts to map the application to every cluster, and explores task-to-core mapping at $f_j>=MA{F}_j^{LC}$ until a feasible task mapping is obtained (as shown in Figure 3d, in blue, recalling ❻). The GM compares the resulting $E_{sys}^{dynamic}$ (which can include $E_{overhead}^{migration}$) of different application-to-cluster mappings for each application, and takes the mapping solution leading to the minimum energy value. The GM decision is highly dependent on the LM decisions (e.g., $f_j$ and task-to-core mapping). The application-to-cluster mapping configuration (in GM) and cluster $f_j$ (indicating task-to-core mapping, in LM) are involved in a loop (see the red loop of Figure 3b). This indicates that mapping and DVFS are considered holistically (Mapping & DVFS). After the first two configurations are fixed for all active applications in the current use case, the LM switches off idle cores to optimize $E_{sys}^{static}+E_{overhead}^{DPM}$. It means that DMP is separated from both mapping and DVFS configurations (Mapping & DVFS → DPM) in the partially separate strategy. It is worth mentioning that compared to the work of [11], which fixes task-to-core mapping at $f_j=MA{F}_j^{LC}$ (recall ❺), the proposed partially separate strategy allows a larger exploration space of task-to-core mapping at $f_j>=MA{F}_j^{LC}$ (see Figure 3d in blue, recalling ❻).

3.2.3. Holistic: Mapping & DVFS & DPM

Figure 3c illustrates the holistic management strategy, which considers the mutual influence of the three system configurations. In this holistic strategy, the GM maps each application to the cluster leading to the lowest $E_{sys}^{dynamic}+E_{sys}^{static}$ (can include $E_{overhead}^{migration}+E_{overhead}^{DPM}$ in Equation (6)) via a greedy strategy. For each active application, the GM attempts to map the application to every cluster, and explores task-to-core mapping at $MA{F}_j^{LC}<=f_j<=f_{j,max}$. This means that the task-to-core mapping exploration continues even when resource constraints are met. The optimized $f_j$ can be set as $f_j^{*}$ (Figure 3d in purple, recalling ❼) for fewer used cores. This provides more opportunities for switching-off cores to optimize $E_{sys}^{static}$. The GM compares the resulting $E_{sys}^{dynamic}+E_{sys}^{static}$ (which can include $E_{overhead}^{migration}+E_{overhead}^{DPM}$) of different application-to-cluster mappings for each application, and takes the mapping solution leading to the minimum energy value. In the holistic strategy, the GM and LM coordinate with each other to explore all the three system configurations (mapping, DVFS, and DPM) in the same loop (see the inner loop in red). Any change in one configuration triggers an update of the other two configurations (Mapping & DVFS & DPM). In contrast to the separate strategies (fully and partially) which separate dynamic energy optimization and static energy optimization into two steps, the holistic strategy simultaneously optimizes dynamic energy, static energy, and system run-time overheads of the overall system.

4. Experimental Results

4.1. Simulation Setup

All the management strategies are implemented in Python running on a PC with Intel CPU I7-9700 at 3.00 GHz configuration. In the experimental evaluations, the platform model, the application model, and design-time-prepared data are set according to the HHMS work of [11]. First, the heterogeneous cluster-based platforms consider big.LITTLE architectures by using ARM Cortex-A series processors (e.g., ARM-Cortex-A9, A7, A15, and A17 ( The A9, A7, A15, and A17 core types have different characteristics of power consumption (1:0.55:2.25:1.3 with regard to A9) and performance (1:1.25:0.625:0.645, with regard to A9). Additionally, the frequency ranges for the four core types respectively are: 0.4–2.0 GHz, 0.4–1.4 GHz, 0.4–2.0 GHz, and 0.4–1.8 GHz, step = 0.1 GHz)). Two-cluster (A7, A15) and four-cluster (A9, A7, A15, A17) platforms are considered, and each cluster contains four or eight cores. Second, as in [11], 10 applications (derived from H263-encoder and H263-decoder and JPEG-decoder with different periods) are captured as an SDF model [24]. All the possible (i.e., 1023) use cases of the 10 considered applications are generated in random sequences, and the use cases have different numbers of active applications. Third, the design-time data (for run-time configurations in HHMS) is also prepared according to [11]. For each of the 10 applications, the design-time prepared data includes design-time mappings (using different numbers of cores/threads) and their corresponding MAFs (meeting performance constraint) in the four core types.

Notice that the proposed energy-management strategy in this paper is not dependent on the used power models. For experimental evaluation purposes, this paper uses the dynamic power model $Powe{r}_{dynamic}(cluste{r}_j,f_j)=\xi \times f_j^3$ ($\xi$ is a coefficient that is dependent on the application tasks and the assigned core type) for ARM big.LITTLE architectures [11,25], which implies a linear correlation between $f_j$ and the supply voltage $V_{dd}$. With the generic static power model $Powe{r}_{static}=V_{dd}.I_{leak}(V_{dd},T)$ [22] ($I_{leak}$ is the leakage current, which depends on the supply voltage and the cores’ temperature T), this paper converts the static power model into a frequency-squared ($Powe{r}_{static}\propto f^2$) relationship. This static model may not be perfectly accurate, but it roughly fits the real data measured on the ODROID XU3 platform [1] with a coarse-grained modeling process. Taking into account the variation of different platforms (different technology nodes), a frequency-linear model ($Powe{r}_{static}\propto f$) will be also discussed in the experiment.

This paper introduces the static energy ratio (SER) to give an intuition of the distribution of dynamic and static power/energy in the system. SER refers to the ratio of static energy compared to dynamic energy (i.e., $SER=\frac{E_{sys}^{static}}{E_{sys}^{dynamic}}$) for 10 applications executing in each cluster at 1GHz. The coefficient (e.g., $\beta$) of the used static power model (e.g., $Powe{r}_{static}=\beta \times f^2$) are set for different clusters (core types) according to the SER. In the experimental evaluations, SER is set to 0.02 (or 0.05, 0.1, or 0.2), meaning that static energy is 2% (or 5%, 10%, or 20%) of dynamic energy at 1GHz. Different SERs are used to evaluate the effectiveness of proposed management strategies on platforms with different power/energy characteristics.

Evaluated Hybrid-Hierarchical Management Strategies

This section introduces the management strategies that are compared in the experiments. All the considered management strategies are based on the same design-time-prepared data (e.g., design-time, task-to-core mappings and MAFs). The fully separate, partially separate, and holistic management strategies have been illustrated in Section 3.2. To compare the energy consumption, resource utilization, and complexity of three management strategies, exhaustive and genetic strategies are used as baselines.

• Exhaustive: The exhaustive strategy explores all the possible system configurations, i.e., application-to-cluster mappings, task-to-core mappings (indicating DVFS or cluster frequency, see Section 2), and DPM decisions. It aims to achieve the minimum energy (including $E_{sys}^{dynamic}$, $E_{sys}^{static}$, $E_{overhead}^{DPM}$, and $E_{overhead}^{migration}$) of the overall system, satisfying application performance (e.g., period) constraints and resource constraints.
• Genetic: The genetic algorithm is based on a standard genetic algorithm [26]. It initializes population individuals, which consist of a set of random chromosomes. Each chromosome consists of two genes, indicating application-to-cluster mapping and task-to-core mapping selection respectively. The fitness function is defined by $max(1/\sum E_{sys})$ (see Equation (6), $E_{sys}$ is the system energy in a use case).

The HHMS-based management strategies are compared in special and general cases.

4.2. Special Case

In the special case, it is assumed that each use case starts on an empty platform, which indicates that the use cases are independent. All the active applications (in a use case) are considered for mapping, DVFS, and DPM configurations. Intuitively, energy-saving benefits (due to good system configurations) increase with execution time, whereas run-time overheads (e.g., $E_{overhead}^{DPM}$, $E_{overhead}^{migration}$; recall Section 3) depend on DPM and mapping/migration decisions (rather than system execution time). If the system execution time is long enough (as in the special case), the run-time overhead can be neglected. In the special case, each use case is assumed to be active for a long enough time (e.g., 100 least common multiple (LCM) of the active application periods in each use case) to neglect possible run-time overheads.

4.2.1. Evaluation of Different Platform Sizes

To gain a better insight into the heterogeneous cluster-based systems, this experiment compares the energy-optimal solutions (obtained by the exhaustive strategy) due to different platform sizes (#clusters × #cores, e.g., 2 × 4, 2 × 8, 4 × 4, 4 × 8), different SER (e.g., 0.02, 0.1, 0.2), and different static power models (e.g., $Powe{r}_{static}\propto f^2$ and $Powe{r}_{static}\propto f$). Figure 4 shows the comparison results of overall energy ($E_{sys}^{dynamic}+E_{sys}^{static}$) and the total number of used cores ($N_{all}^{used}$) for a use case (on average, normalized to 2 × 4 and SER = 0.02). Notice that there can be infeasible solutions for a big use case (e.g., >8 applications) on a small platform size, such as 2 × 4. For a fair comparison, only use cases (1002 use cases) that are feasible for all the 4 platform sizes are considered.

Three main observations can be made from Figure 4. First, considering different platform sizes with fixed SER and fixed static power model, $E_{sys}$ decreases with increasing platform sizes (#cluster × #cores). The 2 × 4 platform has the highest $E_{sys}$, followed by 2 × 8, 4 × 4, and 4 × 8 platforms. This is the result of $E_{sys}^{dynamic}$ and $E_{sys}^{static}$ tradeoffs.

• $E_{sys}^{dynamic}$ dominates the overall energy of each platform (see Figure 4a). As illustrated by Equation (2), $E_{sys}^{dynamic}$ is highly dependent on core types and cluster frequency, i.e., selected MAF in each cluster. Generally speaking, more clusters/cores can lead to lower $E_{sys}^{dynamic}$. This allows applications with different MAFs to be separated into more different clusters, and lower cluster frequencies ($f_j$) can be achieved for lower $E_{sys}^{dynamic}$. Particularly for each cluster, more available cores allow more threads (of design-time mapping) to be selected for each application, and can consequently reduce the common MAF ($f_j$, recall Figure 2). This observation is aligned to the work of [11].
• $E_{sys}^{static}$ is highly dependent on the number of used cores ($N_{all}^{used}$) and $f_j$ (considering the used static power model). A 2 × 4 platform has the highest static energy due to the high $f_j$ (reflected by $E_{sys}^{dynamic}$). A 2 × 8 platform has the second highest static energy due to the large $N_{all}^{used}$ (see Figure 4b).

Secondly, on the same platform size and same static power model (e.g., $Powe{r}_{static}\propto f^2$) when SER grows from 0.02 to 0.2, $E_{sys}^{dynamic}$ slightly increases (SER = 0.02 vs. SER = 0.2). $E_{sys}^{static}$ significantly increases, whereas $N_{all}^{used}$ decreases. This is because, with growing SER, optimizing $E_{sys}^{static}$ becomes more important for the overall energy optimization. The exhaustive strategy puts more effort into $E_{sys}^{static}$ optimization (by reducing $N_{all}^{used}$), and reduces the effort in $E_{sys}^{dynamic}$ optimization (by increasing $N_{all}^{used}$ for more threads in each cluster). This indicates that the optimal configuration of mapping, DVFS, and DPM is also dependent on SER characteristic of platforms.

Thirdly, comparing the results of $Powe{r}_{static}\propto f^2$ with $Powe{r}_{static}\propto f$ models, one significant difference lies in $N_{all}^{used}$. Mathematically, the latter model is less dependent on cluster frequency, which means it is more dependent on $N_{all}^{used}$. Thus, as SER grows, the exhaustive strategy puts more effort into reducing $N_{all}^{used}$ (with regard to the former static energy model; see the slopes of Figure 4b).

4.2.2. Evaluation of Different Management Strategies

This experiment evaluates the proposed management strategies (e.g., fully separate, partially separate, and holistic) on different platform sizes and different SERs (by using the $Powe{r}_{static}\propto f^2$ model). Figure 5 shows $E_{sys}$ ($E_{sys}^{dynamic}+E_{sys}^{static}$) and $N_{all}^{used}$. This experiment uses exhaustive and genetic strategies as the comparison baseline, and all results are normalized to the exhaustive results when SER = 0.02.

As shown in Figure 5, the exhaustive method achieves the energy-optimal solutions. Notice that the exhaustive strategy is not necessary leading to the smallest $N_{all}^{used}$, as it puts $E_{sys}$ optimization in the highest priorities. The genetic strategy achieves the closest $E_{sys}$ to the exhaustive strategy (up to 12.4% difference) on the 2 × 4 platform. However, genetic behavior gets worse as the platform sizes increase (e.g., especially on 4 × 8 platform). Because big platform sizes lead to increasing exploration space, it is difficult for the genetic strategy to explore optimized solutions within a reasonable time (for the run-time system). Moreover, the fully separate (FS in Figure 5) strategy leads to the highest $E_{sys}$ (up to 42.1% difference to exhaustive). This strategy uses LPF [4] for application-to-cluster mapping (recall Section 3.2.1), which gives the highest mapping priority to the lowest power cluster. As most active applications (in a use case) are assigned to one cluster, the cluster frequency becomes higher (due to higher common MAF; recall Section 2.2). This can lead to a large increase in $E_{sys}^{dynamic}$ as well as $E_{sys}$.

In contrast, both partially separate (PS in Figure 5) and holistic (H in Figure 5) strategies achieve close $E_{sys}$ to exhaustive (e.g., about 20% difference on 2 × 4 platform, and a 5% difference on 4 × 8 platform). The partially separate strategy achieves better energy optimization on two-cluster platforms (2 × 4 and 2 × 8). This strategy first optimizes $E_{sys}^{dynamic}$ (setting low-cluster frequencies, via DVFS) then optimizes $E_{sys}^{static}$ (via DPM). This indicates that $E_{sys}^{dynamic}$ optimization is a higher priority than $E_{sys}^{static}$ optimization on small platform sizes (e.g., 2 clusters). On the other hand, the holistic strategy achieves lower energy on four-cluster platforms (4 × 4 and 4 × 8). It reduces the importance of $E_{sys}^{dynamic}$ optimization (with regard to the partially separate strategy). This explains why its $E_{sys}^{dynamic}$ is higher than that of the partially separate strategy (see Figure 5(a.1–a.4)). It can be concluded that $E_{sys}^{static}$ optimization becomes as important as $E_{sys}^{dynamic}$ on big platform sizes (e.g., four clusters). Additionally, when SER = 0.02, the partially separate strategy is always more energy efficient than the holistic strategy. Low SER means $E_{sys}^{dynamic}$ dominating $E_{sys}$, reduction in cluster frequency is more effective than DPM to optimize the overall energy. Lastly, the holistic strategy uses the least $N_{all}^{used}$, indicating that this strategy is the most resource efficient.

Similar observations can be seen for the $Powe{r}_{static}\propto f$ model. Because the static power model shows lower dependency (with regard to $Powe{r}_{static}\propto f^2$), it has higher dependency on $N_{all}^{used}$. A steeper slope can be seen from the evolution of $N_{all}^{used}$ on SER (recall Section 4.2.1), which is not specifically shown here due to the space limit of the paper.

4.3. General Case

In the general case, a use case can start on a non-empty platform (with existing applications). Ideally, only the newly active applications (with regard to the previous use case) are considered (in terms of mapping, DVFS, and DPM) in each use case. To further optimize the energy consumption of the overall system, old existing applications (that have been active since the previous use case) can be allowed to migrate from one cluster to another. It means that the three system configurations in each use case is dependent on its previous use case. In contrast to the special case (Section 4.2), the general case considers both long and short execution time of the system. With short (e.g., #LCM < 100) system execution time, it is important to trade off energy savings and run-time overheads (e.g., $E_{overhead}^{DPM}$, $E_{overhead}^{migration}$; recall Section 3). Notice that the exhaustive and genetic strategies are not considered in the general case, due to the huge design space and long exploration time of the two strategies.

4.3.1. Evaluation of Different Management Strategies without Migration

This experiment evaluates the proposed management strategies that prohibit migrating applications across clusters. Figure 6 shows $E_{sys}$ ($E_{sys}^{dynamic}+E_{sys}^{static}+E_{overhead}^{DPM}$) and $N_{all}^{used}$, for different platform sizes and different execution durations (#LCM = 1, 10, 100, 1000). Note that the different policies denoted by “-M0” mean that they allow 0 migration. The experimental results are normalized to exhaustive -M0 when #LCM = 1.

As shown in Figure 6, the observations in Section 4.2.2 (the special case) still hold when the general case allows 0 migration (across clusters). (1) Fully separate (FS-M0 in Figure 6) results in the highest $E_{sys}$ as well as the most $N_{all}^{used}$. This indicates that giving clusters different mapping priorities is neither energy efficient nor resource efficient (recall Section 3.2.1). (2) The partially separate (PS-M0 in Figure 6) and holistic (H-M0 in Figure 6) strategies have better energy efficiency in two-cluster platforms (2 × 4, 2 × 8) and in four-cluster platforms (4 × 4, 4 × 8), respectively. The partially separate (PS-M0) strategy gives higher priority to minimizing cluster frequency (recall Section 3.2.2). This indicates that $E_{sys}^{dynamic}$ optimization (via DVFS) is in higher priority than $E_{sys}^{static}$ optimization (via DPM) on small platform sizes (e.g., two clusters). (3) The holistic (H-M0) strategy leads to the most resource-efficient solutions (see Figure 6(b.1–b.4)). Lastly, with the increases in execution time (#LCM = 1, 10, 100, 1000), $E_{sys}^{dynamic}$ and $E_{sys}^{static}$ keep increasing, whereas $E_{overhead}^{DPM}$ becomes negligible when execution time becomes large (e.g., #LCM > 10).

4.3.2. Evaluation of Different Management Strategies Allowing Migrations

To illustrate the benefit of allowing application migration across clusters, this experiment evaluates the partially separate and holistic strategies with 0,1,2 migrations (denoted with -M0,1,2). The fully separate strategy is not considered here due to its high energy consumption (see Section 3.2.1 and Section 3.2.2). Figure 7 shows $E_{sys}$ ($E_{sys}^{dynamic}+E_{sys}^{static}+E_{overhead}^{DPM}+E_{overhead}^{migration}$), $N_{all}^{used}$, and the actual number of migrations (#migration), for different platform sizes and different execution times (#LCM = 1, 10, 100). The experimental results are normalized to holistic-M0 (#LCM = 1). Notice that because even the two considered strategies allow one or two old existing applications (in the previous use case) to be migrated to new clusters, the actual #migration can be smaller than one (or two, on average per use case). This is because application migration does not necessarily lead to energy saving. This is a tradeoff between energy savings and migration overhead.

As shown in Figure 7, migration can generally lead to energy saving within long execution time (#LCM > 10). With increasing #LCM, the run-time overheads ($E_{run-time}^{DPM}+E_{run-time}^{migration}$) become negligible compared to energy saving. This can be regarded as approaching the special case (see Section 4.2). However, when the execution time is small (#LCM = 1), migration can lead to an increase in $E_{sys}$ (e.g., Figure 7(a.3)). This indicates that frequent migrations in subsequent use cases (i.e., 1023 use cases in this experiment) are not energy efficient. Additionally, regarding the partially separate (PS-M0,1,2 in Figure 7) and holistic strategies (H-M0,1,2 in Figure 7), they lead to better energy efficiency in the 2 × 4 platform and 4-cluster platforms (4 × 4, 4 × 8), respectively. The two strategies lead to comparable $E_{sys}$ in 2 × 8 platform. This is also aligned with the previous observation (in Section 4.2.2 and Section 4.3.1) that minimization of cluster frequency (for $E_{sys}^{dynamic}$ optimization) is a higher priority than DPM ($E_{sys}^{static}$ optimization) in small platforms (e.g., 2 × 4). Moreover, it can be observed that the holistic strategy (H-M0,1,2) results in fewer $N_{all}^{used}$ and fewer migrations than the partially separate strategy (PS-M0,1,2).

4.4. Experimental Discussion

This section discusses the characteristics of different management strategies from the experimental observations. Before such discussions,

Table 3. Time (ms) to explore a use-case solution (averaged in 1023 use cases) of different strategies.

Platform	Special Case					General Case
						-M0				-M1		-M2
	Exhaustive	Genetic	FS	PS	H	Exhaustive	FS	PS	H	PS	H	PS	H
2 × 4	1.35	804.35	0.05	0.12	0.12	0.23	0.07	0.13	0.14	0.15	0.16	0.16	0.18
2 × 8	1.54	40.93	0.05	0.12	0.12	0.27	0.08	0.12	0.13	0.15	0.15	0.15	0.16
4 × 4	282.99	46.15	0.06	0.18	0.18	7.69	0.08	0.16	0.17	0.19	0.19	0.20	0.22
4 × 8	313.98	31.20	0.06	0.18	0.19	8.29	0.09	0.16	0.17	0.19	0.20	0.21	0.22

FS: fully separate, PS: partially separate, H: holistic, -M0,1,2: allowing 0,1,2 migration(s) across clusters.

summarizes the exploration time (time (ms) to explore a use case solution, on average) to characterize strategy complexity. Compared to the special case, the general case has smaller exploration time. This is because the general case mainly considers applications that are newly activated at run time (with regard to previous use cases) to reduce their exploration burden. Particularly for the general case, allowing migration (-M1,2 vs. -M0) leads to an increase in exploration time.

In summary, the experimental observations derive insights into the characteristics of heterogeneous cluster-based platforms (supporting per-cluster DVFS and per-core DPM) and different management (implemented in HHMS manner), as follows.

• Larger platform sizes (with more available clusters/cores) can lead to better energy efficiency. Dynamic energy is highly dependent on cluster frequency levels, which are determined by application-to-cluster mapping and task-to-core mapping (reflected by Equations (1) and (2)). Static energy is related to cluster frequency model (depending on the static power model) and the number of used cores.
• The fully separate strategy gives different mapping priorities to clusters. The resulting high frequencies (of some clusters) can lead to high energy consumption of the overall system. This strategy has lowest strategy complexity (see Table 3) due to its simplicity.
• The partially separate and holistic strategies lead to similar energy consumption (5–20% difference with regard to exhaustive) with low strategy complexity (up to 1652× lower with regard to exhaustive).

  –

  The partially separate strategy gives higher priority into dynamic energy optimization (with low frequency). It can achieve better energy efficiency than the holistic strategy on two-cluster platforms. This indicates that dynamic energy optimization is more important (with regard to static energy optimization) on small platforms.

  –

  The holistic strategy optimizes dynamic and static energy simultaneously, and it achieves better energy efficiency in four-cluster platforms. The holistic strategy has the best resource efficiency, i.e., using the least cores for energy optimization. Compared to the partially separate strategy, the holistic strategy has slightly higher complexity due to its larger exploration space (see Section 3.2).

5. Conclusions

For multiple applications concurrently executing on heterogeneous cluster-based platforms (e.g., ARM big.LITTLE), this paper formulates the dependencies between application-to-cluster mapping, task-to-core mapping, per-cluster DVFS, and per-core DPM configurations into a 0–1 ILP model to optimize overall energy (including run-time overhead of DPM setting and application migration). Based on the 0–1 ILP model and hybrid hierarchical management structure, three heuristic (or greedy) management strategies (fully separate, partially separate, and holistic) are compared. The experimental evaluations are performed in 1023 use cases (generated in random orders) on different platform sizes (e.g., with 2 and 4 clusters), showing that the fully separate and holistic strategies can lead to good energy efficiency (5–20% difference with regard to exhaustive) and low strategy complexity (up to 1652× lower with regard to exhaustive).

The proposed fully separate and holistic management strategies are expected to be applied to achieve energy efficiency for cluster-based platforms, such as embedded mobile platforms (e.g., ODROID XU3/4 [1]) and high-performance computing (HPC) platforms (e.g., ODROID-MC1 [2]) that support per-cluster DVFS. Particularly for HPC applications, their tasks are commonly used to represent kernels, and each kernel consists of one or multiple tasks. The kernel execution can be regarded as design/run-time mappings by using numbers of cores/threads in our proposed strategies. This shows the potential to use the proposed strategies in the HPC domains. To have further validations on embedded systems and HPC, our future work plans include: (1) validating the proposed strategies based on the real implementation by using multi-threaded PARSEC [27] on ODROID XU4 and ODROID-MC1 platforms, and (2) targeting machine-learning predictions in order to avoid the efforts of design-time preparation (MAFs for different threads).