Cost-Aware Bandits for Efficient Channel Selection in Hybrid Band Networks
Abstract
:1. Introduction
1.1. Related Work
1.2. Paper Contributions
- We aim to relax/reformulate the HBS multi-objective optimization problem and obtain acceptable solutions (i.e., sub-optimal HBS decisions) in real time without prior channel knowledge using online learning techniques that easily handle blockage and energy consumption during the selection process.
- We reformulate the HBS optimization problem into a cost subsidy multi-armed bandit (CS-MAB) that accounts for the cost during selection in both exploitation and exploration terms.
- We propose CS—upper confidence bound (CSUCB-HBS) and CS—Thompson sampling (CSTS-HBS) algorithms and evaluate their performance compared with the ordinary MAB techniques (UCB and TS), and traditional HBS (i.e., conventional and random choice).
- Simulation results indicate the superior performance of our proposed CS-MAB techniques over classical MAB methods, especially CSTS-HBS, which exhibits better performance than CSUCB-HBS and others.
2. System Model
3. Problem Formulation
4. Envisioned CS-HBS Methods
4.1. CSUCB-HBS Algorithm
4.2. CSTS-HBS Algorithm
Algorithm 1: CSUCB/CSTS-HBS Algorithms |
5. Results
6. Conclusions and Outlook
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Acknowledgments
Conflicts of Interest
References
- Abuella, H.; Elamassie, M.; Uysal, M.; Xu, Z.; Serpedin, E.; Qaraqe, K.A.; Ekin, S. Hybrid RF/VLC Systems: A Comprehensive Survey on Network Topologies, Performance Analyses, Applications, and Future Directions. IEEE Access 2021, 9, 160402–160436. [Google Scholar] [CrossRef]
- Chen, Y.; Ai, B.; Niu, Y.; He, R.; Zhong, Z.; Han, Z. Resource Allocation for Device-to-Device Communications in Multi-Cell Multi-Band Heterogeneous Cellular Networks. IEEE Trans. Veh. Technol. 2019, 68, 4760–4773. [Google Scholar] [CrossRef] [Green Version]
- Mughal, B.; Fadlullah, Z.M.; Fouda, M.M.; Ikki, S. Allocation Schemes for Relay Communications: A Multiband Multichannel Approach Using Game Theory. IEEE Sens. Lett. 2022, 6, 7500104. [Google Scholar] [CrossRef]
- Najla, M.; Mach, P.; Becvar, Z. Deep Learning for Selection Between RF and VLC Bands in Device-to-Device Communication. IEEE Wirel. Commun. Lett. 2020, 9, 1763–1767. [Google Scholar] [CrossRef]
- Sakib, S.; Tazrin, T.; Fouda, M.M.; Fadlullah, Z.M.; Nasser, N. A Deep Learning Method for Predictive Channel Assignment in Beyond 5G Networks. IEEE Netw. 2021, 35, 266–272. [Google Scholar] [CrossRef]
- Shrivastava, S.; Chen, B.; Chen, C.; Wang, H.; Dai, M. Deep Q-Network Learning Based Downlink Resource Allocation for Hybrid RF/VLC Systems. IEEE Access 2020, 8, 149412–149434. [Google Scholar] [CrossRef]
- Sakib, S.; Tazrin, T.; Fouda, M.M.; Fadlullah, Z.M.; Nasser, N. An Efficient and Light-weight Predictive Channel Assignment Scheme for Multi-Band B5G Enabled Massive IoT: A Deep Learning Approach. IEEE Internet Things J. 2021, 8, 5285–5297. [Google Scholar] [CrossRef]
- Bakri, S.; Brik, B.; Ksentini, A. On using reinforcement learning for network slice admission control in 5G: Offline vs. online. Int. J. Commun. Syst. 2021, 34, 1987–2007. [Google Scholar] [CrossRef]
- Nasir, Y.S.; Guo, D. Multi-Agent Deep Reinforcement Learning for Dynamic Power Allocation in Wireless Networks. IEEE J. Sel. Areas Commun. 2019, 37, 2239–2250. [Google Scholar] [CrossRef] [Green Version]
- Wang, S.; Lv, T. Dynamic Multichannel Access for 5G and Beyond with Fast Time-Varying Channel. In Proceedings of the ICC 2020—2020 IEEE International Conference on Communications (ICC), Dublin, Ireland, 7–11 June 2020; pp. 1–6. [Google Scholar] [CrossRef]
- Wang, Z.; Zhang, T.; Liu, Y.; Xu, W. Caching Placement and Resource Allocation for AR Application in UAV NOMA Networks. In Proceedings of the GLOBECOM 2020—2020 IEEE Global Communications Conference, Taipei, Taiwan, 7–11 December 2020; pp. 1–6. [Google Scholar] [CrossRef]
- Hashima, S.; Fadlullah, Z.M.; Fouda, M.M.; Mohamed, E.M.; Hatano, K.; ElHalawany, B.M.; Guizani, M. On Softwarization of Intelligence in 6G Networks for Ultra-Fast Optimal Policy Selection: Challenges and Opportunities. IEEE Netw. 2022, 1–9. [Google Scholar] [CrossRef]
- Yin, R.; Wu, Z.; Liu, S.; Wu, C.; Yuan, J.; Chen, X. Decentralized Radio Resource Adaptation in D2D-U Networks. IEEE Internet Things J. 2021, 8, 6720–6732. [Google Scholar] [CrossRef]
- Du, Z.; Wang, C.; Sun, Y.; Wu, G. Context-Aware Indoor VLC/RF Heterogeneous Network Selection: Reinforcement Learning With Knowledge Transfer. IEEE Access 2018, 6, 33275–33284. [Google Scholar] [CrossRef]
- Wang, C.; Wu, G.; Du, Z.; Jiang, B. Reinforcement learning based network selection for hybrid VLC and RF systems. In MATEC Web of Conferences; EDP Sciences: Les Ulis, France, 2018; Volume 173, p. 03014. [Google Scholar] [CrossRef] [Green Version]
- Lattimore, T. Bandit Algorithms; Cambridge University Press: Cambridge, UK, 2020. [Google Scholar]
- Hashima, S.; ElHalawany, B.M.; Hatano, K.; Wu, K.; Mohamed, E.M. Leveraging Machine-Learning for D2D Communications in 5G/Beyond 5G Networks. Electronics 2021, 10, 169. [Google Scholar] [CrossRef]
- Hashima, S.; Hatano, K.; Takimoto, E.; Mohamed, E.M. Neighbor Discovery and Selection in Millimeter Wave D2D Networks Using Stochastic MAB. IEEE Commun. Lett. 2020, 24, 1840–1844. [Google Scholar] [CrossRef]
- Hashima, S.; Hatano, K.; Kasban, H.; Mohamed, E.M. Wi-Fi Assisted Contextual Multi-Armed Bandit for Neighbor Discovery and Selection in Millimeter Wave Device to Device Communications. Sensors 2021, 21, 2835. [Google Scholar] [CrossRef]
- Hashima, S.; Hatano, K.; Takimoto, E.; Mohamed, E.M. Minimax Optimal Stochastic Strategy (MOSS) For Neighbor Discovery and Selection In Millimeter Wave D2D Networks. In Proceedings of the 2020 23rd International Symposium on Wireless Personal Multimedia Communications (WPMC), Okayama, Japan, 19–26 October 2020; pp. 1–6. [Google Scholar] [CrossRef]
- Mohamed, E.M.; Hashima, S.; Hatano, K.; Aldossari, S.A.; Zareei, M.; Rihan, M. Two-Hop Relay Probing in WiGig Device-to-Device Networks Using Sleeping Contextual Bandits. IEEE Wirel. Commun. Lett. 2021, 10, 1581–1585. [Google Scholar] [CrossRef]
- Mohamed, E.M.; Hashima, S.; Aldosary, A.; Hatano, K.; Abdelghany, M.A. Gateway Selection in Millimeter Wave UAV Wireless Networks Using Multi-Player Multi-Armed Bandit. Sensors 2020, 20, 3947. [Google Scholar] [CrossRef]
- Hashima, S.; Mohamed, E.M.; Hatano, K.; Takimoto, E. WiGig Wireless Sensor Selection Using Sophisticated Multi Armed Bandit Schemes. In Proceedings of the 2021 Thirteenth International Conference on Mobile Computing and Ubiquitous Network (ICMU), Tokyo, Japan, 17–19 November 2021; pp. 1–6. [Google Scholar] [CrossRef]
- Barrachina-Muñoz, S.; Chiumento, A.; Bellalta, B. Multi-Armed Bandits for Spectrum Allocation in Multi-Agent Channel Bonding WLANs. IEEE Access 2021, 9, 133472–133490. [Google Scholar] [CrossRef]
- Zuo, J.; Joe-Wong, C. Combinatorial Multi-armed Bandits for Resource Allocation. In Proceedings of the 2021 55th Annual Conference on Information Sciences and Systems (CISS), Baltimore, MD, USA, 24–26 March 2021. [Google Scholar] [CrossRef]
- Mohamed, E.M.; Hashima, S.; Hatano, K. Energy Aware Multi-Armed Bandit for Millimeter Wave Based UAV Mounted RIS Networks. IEEE Wirel. Commun. Lett. 2022. [Google Scholar] [CrossRef]
- Mohamed, E.M.; Hashima, S.; Hatano, K.; Aldossari, S.A. Two-Stage Multiarmed Bandit for Reconfigurable Intelligent Surface Aided Millimeter Wave Communications. Sensors 2022, 22, 2179. [Google Scholar] [CrossRef]
- Mohamed, E.M.; Hashima, S.; Hatano, K.; Kasban, H.; Rihan, M. Millimeter-Wave Concurrent Beamforming: A Multi-Player Multi-Armed Bandit Approach. Comput. Mater. Contin. 2020, 65, 1987–2007. [Google Scholar] [CrossRef]
- ElHalawany, B.M.; Hashima, S.; Hatano, K.; Wu, K.; Mohamed, E.M. Leveraging Machine Learning for Millimeter Wave Beamforming in Beyond 5G Networks. IEEE Syst. J. 2021, 1–12. [Google Scholar] [CrossRef]
- Hashima, S.; Hatano, K.; Kasban, H.; Rihan, M.; Mohamed, E.M. Multiagent Multi-Armed Bandit Techniques for Millimeter Wave Concurrent Beamforming. In Proceedings of the 2020 8th International Japan-Africa Conference on Electronics, Communications, and Computations (JAC-ECC), Virtual, 14–15 December 2020; pp. 56–59. [Google Scholar] [CrossRef]
- Fouda, M.; Hashima, S.; Sakib, S.; Fadlullah, Z.; Hatano, K.; Shen, X. Optimal Channel Selection in Hybrid RF/VLC Networks: A Multi-Armed Bandit Approach. IEEE Trans. Veh. Technol. 2022. [Google Scholar] [CrossRef]
- Hashima, S.; Fouda, M.M.; Sakib, S.; Fadlullah, Z.M.; Hatano, K.; Mohamed, E.M.; Shen, X. Energy-Aware Hybrid RF-VLC Multi-Band Selection in D2D Communication: A Stochastic Multi-Armed Bandit Approach. IEEE Internet Things J. 2022. [Google Scholar] [CrossRef]
- Hashima, S.; Fouda, M.M.; Fadlullah, Z.M.; Mohamed, E.M.; Hatano, K. Improved UCB-based Energy-Efficient Channel Selection in Hybrid-Band Wireless Communication. In Proceedings of the 2021 IEEE Global Communications Conference (GLOBECOM), Madrid, Spain, 7–11 December 2021; pp. 1–6. [Google Scholar] [CrossRef]
- Wei, N.; Lin, X.; Zhang, Z. Optimal Relay Probing in Millimeter-Wave Cellular Systems with Device-to-Device Relaying. IEEE Trans. Veh. Technol. 2016, 65, 10218–10222. [Google Scholar] [CrossRef] [Green Version]
- Boban, M.; Dupleich, D.; Iqbal, N.; Luo, J.; Schneider, C.; Müller, R.; Yu, Z.; Steer, D.; Jämsä, T.; Li, J.; et al. Multi-Band Vehicle-to-Vehicle Channel Characterization in the Presence of Vehicle Blockage. IEEE Access 2019, 7, 9724–9735. [Google Scholar] [CrossRef]
Simulation Parameters | Value | |
---|---|---|
Number of channels | 4 (WiFi 2.4 GHz, 5.25 GHz, WiGig 38 GHz, VLC GHz) | |
T, , | 1000, 1 TB, 1% | |
Operating frequencies of each channel | 5.25, 2.4, 38, GHz | |
40, 20, 40, 20 MHz | ||
r | {10–100} m | |
Blocking model [35] | Small blocker: {length, width, height} | {5.07, 1.69, 1.93} m |
Large blocker: {length, width, height} | {7.01, 2.04, 2.63} m |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Hashima, S.; Hatano, K.; Fouda, M.M.; Fadlullah, Z.M.; Mohamed, E.M. Cost-Aware Bandits for Efficient Channel Selection in Hybrid Band Networks. Electronics 2022, 11, 1782. https://0-doi-org.brum.beds.ac.uk/10.3390/electronics11111782
Hashima S, Hatano K, Fouda MM, Fadlullah ZM, Mohamed EM. Cost-Aware Bandits for Efficient Channel Selection in Hybrid Band Networks. Electronics. 2022; 11(11):1782. https://0-doi-org.brum.beds.ac.uk/10.3390/electronics11111782
Chicago/Turabian StyleHashima, Sherief, Kohei Hatano, Mostafa M. Fouda, Zubair M. Fadlullah, and Ehab Mahmoud Mohamed. 2022. "Cost-Aware Bandits for Efficient Channel Selection in Hybrid Band Networks" Electronics 11, no. 11: 1782. https://0-doi-org.brum.beds.ac.uk/10.3390/electronics11111782