An Effective HS-CG Update: Avoiding Oscillations in Nonlinear Optimization
Article Sidebar
Main Article Content
Abstract
In this research, we introduce a modified direction update formula to improve the descent features of iterative optimization, therefore presenting a fresh approach to the Conjugate Gradient (CG) method. The proposed method adjusts the search direction at each iteration by incorporating gradient and step projections, weighted by inner products between gradient and step vectors. The modified HS-CG approach seeks to decrease the computational cost often associated with conventional approaches and speed up convergence by carefully balancing these projections. Experimental results demonstrate that our approach outperforms standard CG algorithms in achieving faster convergence on a range of benchmark problems, especially in high-dimensional spaces. This enhancement makes the method particularly promising for large-scale optimization challenges encountered in fields such as machine learning and engineering design.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
Tikrit Journal of Pure Science is licensed under the Creative Commons Attribution 4.0 International License, which allows users to copy, create extracts, abstracts, and new works from the article, alter and revise the article, and make commercial use of the article (including reuse and/or resale of the article by commercial entities), provided the user gives appropriate credit (with a link to the formal publication through the relevant DOI), provides a link to the license, indicates if changes were made, and the licensor is not represented as endorsing the use made of the work. The authors hold the copyright for their published work on the Tikrit J. Pure Sci. website, while Tikrit J. Pure Sci. is responsible for appreciate citation of their work, which is released under CC-BY-4.0, enabling the unrestricted use, distribution, and reproduction of an article in any medium, provided that the original work is properly cited.
References
1. Halil IH, Abbo KK, Ebrahim HH, editors. Modifications of Hestenes and Stiefel CG method for solving unconstrained optimization problems. 2021 7th International Conference on Contemporary Information Technology and Mathematics (ICCITM); 2021: IEEE. https://doi.org/10.1109/ICCITM53167.2021.9677756
2. Zhou J, Qiu Y, Zhu S, Armaghani DJ, Li C, Nguyen H, et al. Optimization of support vector machine through the use of metaheuristic algorithms in forecasting TBM advance rate. Engineering Applications of Artificial Intelligence. 2021;97:104015. https://doi.org/10.1016/j.engappai.2020.104015
3. Abdollahzadeh B, Gharehchopogh FS. A multi-objective optimization algorithm for feature selection problems. Engineering with Computers. 2022;38 (Suppl 3):1845-63.
https://doi.org/10.1007/s00366-021-01369-9
4. Andrei N. Nonlinear conjugate gradient methods for unconstrained optimization: Springer; 2020.
5. Hassan Ibrahim A, Kumam P, Abubakar AB, Abubakar J, Muhammad AB. Least-square-based three-term conjugate gradient projection method for ℓ 1-norm problems with application to compressed sensing. Mathematics. 2020;8(4):602.
https://doi.org/10.3390/math8040602
6. Yuan G, Lu J, Wang Z. The PRP conjugate gradient algorithm with a modified WWP line search and its application in the image restoration problems. Applied Numerical Mathematics. 2020;152:1-11. https://doi.org/10.1016/j.apnum.2020.01.019
7. Hassan BA, Dahawi HO, Younus AS. A new kind of parameter conjugate gradient for unconstrained optimization. Indonesian Journal of Electrical Engineering and Computer Science. 2020;17(1):404. https://doi.org/10.11591/ijeecs.v17.i1.pp404-411
8. Khudhur HM, Fawze AAM. An improved conjugate gradient method for solving unconstrained optimisation and image restoration problems. International Journal of Mathematical Modelling and Numerical Optimisation. 2023;13(3):313-25. https://doi.org/10.1504/IJMMNO.2023.132286
9. Ji Z, Dudík M, Schapire RE, Telgarsky M, editors. Gradient descent follows the regularization path for general losses. Conference on Learning Theory; 2020: PMLR.
https://doi.org/10.48550/arXiv.2006.11226
10. Fletcher R. Practical methods of optimization: John Wiley & Sons; 2000.
https://doi.org/10.1093/comjnl/7.2.149
11. Dai Y-H, Yuan Y. A nonlinear conjugate gradient method with a strong global convergence property. SIAM Journal on optimization. 1999;10(1):177-82. https://doi.org/10.6028/jres.049.044
12. Hu Y, Storey C. Global convergence result for conjugate gradient methods. Journal of Optimization Theory and Applications. 1991;71(2):399-405. https://doi.org/10.1002/9781118723203
13. Fletcher R, Reeves CM. Function minimization by conjugate gradients. The computer journal. 1964;7(2):149-54. https://doi.org/10.1051/m2an/196903r100351
14. Hestenes MR, Stiefel E. Methods of conjugate gradients for solving linear systems. Journal of research of the National Bureau of Standards. 1952;49(6):409-36. https://doi.org/10.1137/S1052623497318992
15. Polak E, Ribiere G. Note sur la convergence de méthodes de directions conjuguées. Revue française d'informatique et de recherche opérationnelle Série rouge. 1969;3(16):35-43.
https://doi.org/10.1007/BF00939927
16. Hu W, Wu J, Yuan G. Some modified Hestenes-Stiefel conjugate gradient algorithms with application in image restoration. Applied Numerical Mathematics. 2020;158:360-76.
https://doi.org/10.1016/j.apnum.2020.08.009
17. Ibrahem SA, Halil IH, Abdullah SM. Two Algorithms for Solving Unconstrained Global Optimization by Auxiliary Function Method. Tikrit Journal of Pure Science. 2024;29(3):84-9. https://doi.org/10.25130/tjps.v29i3.1604
18. Abdullah ZM, Asker SA. A New Scaled Three-Term Conjugate Gradient Algorithms for Unconstrained Optimization. Tikrit Journal of Pure Science. 2023;28(4):103-10.
https://doi.org/10.25130/tjps.v28i4.1534
19. Awwal AM, Wang L, Kumam P, Mohammad H, Watthayu W. A projection Hestenes–Stiefel method with spectral parameter for nonlinear monotone equations and signal processing. Mathematical and Computational Applications. 2020;25(2):27. https://doi.org/10.3390/mca25020027
20. Taqi AH, Shaker AN. Development Modified Conjugate Gradiente Algorithm. Tikrit Journal of Pure Science. 2015;20(5):185-92. https://doi.org/10.25130/tjps.v20i5.1254
21. Khudhur HM, Halil IH. Noise removal from images using the proposed three-term conjugate gradient algorithm. Компьютерные исследования и моделирование. 2024;16(4):841-53.
https://doi.org/10.20537/2076-7633-2024-16-4-841-853
22. Gould NI, Orban D, Toint PL. CUTEr and SifDec: A constrained and unconstrained testing environment, revisited. ACM Transactions on Mathematical Software (TOMS). 2003;29(4):373-94. https://doi.org/10.1145/962437.962439
23. Andrei N. An unconstrained optimization test functions collection. Adv Model Optim. 2008;10(1):147-61. https://camo.ici.ro/journal/vol10/v10a10.pdf
24. Bongartz I, Conn AR, Gould N, Toint PL. CUTE: Constrained and unconstrained testing environment. ACM Transactions on Mathematical Software (TOMS). 1995;21(1):123-60.
https://doi.org/10.1145/200979.201043
25. Hager WW, Zhang H. A new conjugate gradient method with guaranteed descent and an efficient line search. SIAM Journal on optimization. 2005;16(1):170-92. https://doi.org/10.1137/030601880