Available Issues
Authors:
| M. Naderuzzaman | Department of Computer Science and Engineering, Sonargaon University, Dhaka, Bangladesh |
| Dr. Mohammod Abul Kashem | Department of Computer Science and Engineering, Dhaka University of Engineering and Technology, Gazipur, Bangladesh |
Index Terms:
Combinatorial Optimization, PairWise, Un-uniform Interaction, Interaction level, test suits, Artificial IntelligenceAbstract:
In the software development life cycle, testing plays a crucial role in identifying errors or bugs, ensuring the verification of requirement specifications, design, analysis, coding, and estimating the software's reliability. As software systems grow larger, the size of the test suite typically expands exponentially. However, conducting exhaustive testing is often impractical due to the challenges posed by combinatorial optimization problems, as well as factors such as cost, constraints, and limited resources. To alleviate the burden on software development, it becomes essential to streamline test suites. Generating an optimal number of test cases is imperative for expediting the overall software testing process. Pairwise testing techniques emerge as pivotal in this context, aiming to reduce the size of test suites. Existing literature highlights the effectiveness of varying the number of interactions among input parameters, significantly diminishing the need for extensive test data. Over the past decade, numerous test data generation strategies have been developed, differing in their support for various interaction levels—ranging from the minimum of two (pairwise) to t (t-way), where t can be any value greater than 2. Additionally, various means, such as Artificial Intelligence and Machine Learning, are employed to accelerate the testing process. A comprehensive literature review is crucial for advancing the development of superior test suite generation techniques. Such an examination reveals research gaps that can inspire new approaches from researchers. This paper aims to review prominent pairwise test suite generation techniques, evaluating their strengths and weaknesses. The literature review underscores that many techniques support pairwise interaction, some support t-way interaction, only a few endorse un-uniform interaction, and none accommodates dynamic interactions among input parameters. Notably, the increasing prevalence of Internet of Things (IoT) devices that receive audio and video (metadata) as input parameters lacks adequate test generation techniques supporting metadata. In addition to identifying pros and cons, this paper offers suggestions to guide future researchers in efficiently addressing combinatorial optimization problems and ensuring cost-effectiveness. The objective is to contribute to the evolution of robust techniques for generating test suites, laying the foundation for more effective and comprehensive software testing methodologies.Conclusion:
development, hence is very profitable. Testing is no longer an additional activity but integral part of software development life cycle (SDLD). Emphasizing on the development of techniques to reduce test cases actually helps in the orderly execution of test cases based on the functions or performances of the target (amount of coverage, execution time, and cost). In this paper, a survey was made on the existing test case generation techniques with their internal details. This study reveals the techniques used in selecting the optimal test cases, and a group of previous works was addressed and compared among them. After reading all the selected research in full, the following was concluded: First, from the review of the works, all the similarity was identified and was summarized. Second, from all the existing testing techniques the dissimilarity was identified. Thirdly all the existing testing was analysed for the drawbacks of each techniques was identified and Finally all the findings were summarized in a tabular form to see the research gap at a glance. Moreover the most widely used in determining the precedence of test cases, as well as the execution time and the amount of error coverage are largely identified to use as a measure for evaluating test cases. Observing from the summary table it becomes clear that the use of artificial intelligence and support of meta-data is the future research area in developing test case optimization technique development. Although in NP-hard, it is impossible to develop a mathematical model to generate a polynomial result, our future works also involves in finding a test data generation model.License:
Articles published in OAJEA are licensed under a Creative Commons Attribution 4.0 International License.Cite This Paper:
M.Naderuzzaman,Mohammod Abul Kashem “Combinatorial Test Suit Generation techniques to Identifying Research Gap: A Systematic Review”, Open Access Journal on Engineering Applications (OAJEA), Volume No. 01, Issue No. 01, Page 18-28, July, 2025. https://oajea.hafizlab.com/article/01-01-003Reference:
[1] Ammann P. and Offutt J., Introduction to Software Testing, Cambridge University Press, 2008.
[2] Garousi, Vahid, and Junji Zhi. "A survey of software testing practices in Canada." Journal of Systems and Software 86, no. 5 (2013): 1354-1376.
[3] Lima, Rui, António Miguel Rosado da Cruz, and Jorge Ribeiro. "Artificial intelligence applied to software testing: A literature review." In 2020 15th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1-6. IEEE, 2020.
[4] Y. Cui, L. Li, and S. Yao (2009). A New strategy for pairwise test case generation. 3rd international Symposium on Intelligent Information Technology Application.
[5] Khandakar Rabbi, Rafiqul Islam, QuaziMamun and Mohammed Golam Kaosar (2014). MTTG: An Efficient Technique for Test Data Generation. 8th International Conference on Software, Knowledge, Information Management and Applications.
[6] Yan, J. & Zhang, J. (2008) A Backtracking Search Tool for Constructing Combinatorial Test Suites. Journal of Systems and Software – Elsevier.
[7] Hartman, Alan, and Leonid Raskin. "Problems and algorithms for covering arrays." Discrete Mathematics 284, no. 1-3 (2004): 149-156.
[8] Yu, Linbin, Yu Lei, Raghu N. Kacker, D. Richard Kuhn, and James Lawrence. "Efficient algorithms for t-way test sequence generation." In 2012 IEEE 17th International Conference on Engineering of Complex Computer Systems, pp. 220-229. IEEE, 2012.
[9] Yan, J. & Zhang, J. (2008) A Backtracking Search Tool for Constructing Combinatorial Test Suites. Journal of Systems and Software – Elsevier.
[10] Chateauneuf, M. &Kreher, D. (2002) On the State of Strength-Three Covering Arrays. Journal of Combinatorial Designs.
[11] Colbourn, C. J., Martirosyan, S. S., Mullen, G. L., Shasha, D., Sherwood, G. B. &Yucas, J. L. (2005) Products of Mixed Covering Arrays of Strength Two. Journal of Combinatorial Designs.
[12] Williams, A. W. (2000) Determination of Test Configurations for Pair-wise Interaction Coverage. Proc. of the 13th International Conference on Testing of Communicating Systems.
[13] Hartman, A. &Raskin, L. (2004) Combinatorial Test Services [Online]. [Accessed on March 2015]. Available from: https://www.research.ibm.com/haifa/projects/verification/mdt/papers/CTSUserDocumentation.pdf.
[14] Bach, J. (2004) ALLPAIRS Test Generation Tool, Version 1.2.1. [Online]. [Accessed on March 2015]. Available from: http://www.satisfice.com/tools.shtml.
[15] Ellims, M., Ince, D. &Petre, M. (2008) AETG vs. Man: an Assessment of the Effectiveness of Combinatorial Test Data Generation. UK, in Technical Report, Department of Computing, Faculty of Mathematics and Computing, Open University.
[16] Cohen, M. B., Dwyer, M. B. & Shi, J. (2007) Exploiting Constraint Solving History to Construct Interaction Test Suites. Proc. of the Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION, 2007. UK, IEEE Computer Society.
[17] Yu-Wen, T. &Aldiwan, W. S. (2000) Automating Test Case Generation for the New Generation Mission Software System. Proc. of the IEEE Aerospace Conference.
[18] Cohen, M. B. (2004) Designing Test Suites for Software Interaction Testing. Computer Science. New Zealand, University of Auckland.
[19] T. Shiba, T. Tsuchiya, and T. Kikuno (2004). Using Artificial Life Techniques to Generate Test Cases for Combinatorial Testing. 28th Annual International Computer Software and Applications Conference.
[20] Forbes, M., Lawrence, J., Lei, Y., Kacker, R. N. & Kuhn, D. R. (2008) Refining the In-Parameter-Order Strategy for Constructing Covering Arrays. NIST Journal of Research.
[21] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, J. Lawrence (2007). IPOG: A general strategy for t-way software testing. 14th Annual IEEE International Conference and Workshops on the Engineering and Computer-Based Systems.
[22] Jenny, Available at: http://www.burtleburtle.net/bob/math/. (Last access date: 27th Sep. 2010).
[23] TVG, Available at: http://sourceforge.net/projects/tvg. (Last access date: 27th Sep. 2010).
[24] Hartman, A. &Raskin, L. (2004) Problems and Algorithms for Covering Arrays. Discrete Mathematics-Elsevier.
[25] Kamal Z. Zamli, Mohammad F.J. Klaib, Mohammed I. Younis, Nor Ashidi Mat Isa, Rusli Abdullah (2009). Design and implementation of a t-way test data generation strategy with automated execution tool support. Information Sciences, Elsevier.
[26] Ahmed, B. S, Zamli, K. Z. (2010). PSTG: A T-Way Strategy Adopting Particle Swarm Optimization. Mathematical/Analytical Modelling and Computer Simulation (AMS).
[27] Khandakar Rabbi, Rafiqul Islam, Quazi Mamun and Mohammed Golam Kaosar (2014). MTTG: An Efficient Technique for Test Data Generation. 8th International Conference on Software, Knowledge, Information Management and Applications.
[28] SabiraKhatun, KhandakarFazley Rabbi, CheYahayaYaakub, Mohammad FJ Klaib, Mohammad Masroor Ahmed (2011). PS2Way: An Efficient Pairwise Search Approach for Test Data Generation. Software Engineering and Computer Systems, Springer Berlin Heidelberg.
[29] KhandakarFazley Rabbi, SabiraKhatun, CheYahayaYaakub, Mohammad FJ Klaib (2011). EasyA: Easy and Effective Way to Generate Pairwise Test Data. 2011 Third International Conference on Computational Intelligence, Communication Systems and Networks.
[30] Hourani, Hussam, Ahmad Hammad, and Mohammad Lafi. "The impact of artificial intelligence on software testing." In 2019 IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT), pp. 565-570. IEEE, 2019.
[31] Grindal, Mats, Jeff Offutt, and Sten F. Andler. "Combination testing strategies: a survey." Software Testing, Verification and Reliability 15, no. 3 (2005): 167-199.
[32] Cadar, Cristian, Vijay Ganesh, Peter M. Pawlowski, David L. Dill, and Dawson R. Engler. "EXE: Automatically generating inputs of death." ACM Transactions on Information and System Security (TISSEC) 12, no. 2 (2008): 1-38.
[33] Arcuri, Andrea, and Lionel Briand. "A practical guide for using statistical tests to assess randomized algorithms in software engineering." In Proceedings of the 33rd international conference on software engineering, pp. 1-10. 2011.
[34] Khatun, Sabira, Khandakar Fazley Rabbi, Che Yahaya Yaakub, Mohammad FJ Klaib, and Mohammad Masroor Ahmed. "PS2Way: an efficient pairwise search approach for test data generation." In Software Engineering and Computer Systems: Second International Conference, ICSECS 2011, Kuantan, Pahang, Malaysia, June 27-29, 2011, Proceedings, Part III 2, pp. 99-108. Springer Berlin Heidelberg, 2011.
[35] Rabbi, Khandakar Fazley, Abul Hashem Beg, and Tutut Herawan. "MT2Way: A novel strategy for pair-wise test data generation." In Computational Intelligence and Intelligent Systems: 6th International Symposium, ISICA 2012, Wuhan, China, October 27-28, 2012. Proceedings, pp. 180-191. Springer Berlin Heidelberg, 2012.
[36] Dorigo, Marco, Vittorio Maniezzo, and Alberto Colorni. "The ant system: An autocatalytic optimizing process." (1991).
[37] Rabbi, Khandakar. "Combinatorial testing strategies based on swarm intelligence." (2017). https://researchoutput.csu.edu.au/en/publications/combinatorial-testing-strategies-based-on-swarm-intelligence
Journal Information
| Frequency: Quarterly |
| Submission to First Decision: 7 days |
| Submission to Acceptance: 30 days |
| Accept to Publish: 15 days |
| Article Processing Charges (APCs): None |
