publications

2023

1. ICSE

   Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code

   Pan, Rangeet, Ibrahimzada, Ali Reza,  Krishna, Rahul, Sankar, Divya, Wassi, Lambert Pouguem, Merler, Michele, Sobolev, Boris, Pavuluri, Raju, Sinha, Saurabh, and Jabbarvand, Reyhaneh

   In Proceedings of the 46th International Conference on Software Engineering (ICSE ’24), April 14-19, 2024, Lisbon, Portugal. 2023

   Abs HTML PDF

   Code translation aims to convert source code from one programming language (PL) to another. Given the promising abilities of large language models (LLMs) in code synthesis, researchers are exploring their potential to automate code translation. The prerequisite for advancing the state of LLM-based code translation is to understand their promises and limitations over existing techniques. To that end, we present a large-scale empirical study to investigate the ability of general LLMs and code LLMs for code translation across pairs of different languages, including C, C++, Go, Java, and Python. Our study, which involves the translation of 1,700 code samples from three benchmarks and two real-world projects, reveals that LLMs are yet to be reliably used to automate code translation—with correct translations ranging from 2.1% to 47.3% for the studied LLMs. Further manual investigation of unsuccessful translations identifies 15 categories of translation bugs. We also compare LLM-based code translation with traditional non-LLM-based approaches. Our analysis shows that these two classes of techniques have their own strengths and weaknesses. Finally, insights from our study suggest that providing more context to LLMs during translation can help them produce better results. To that end, we propose a prompt-crafting approach based on the symptoms of erroneous translations; this improves the performance of LLM-based code translation by 5.5% on average. Our study is the first of its kind, in terms of scale and breadth, that provides insights into the current limitations of LLMs in code translation and opportunities for improving them. Our dataset—consisting of 1,700 code samples in five PLs with 10K+ tests, 43K+ translated code, 1,748 manually labeled bugs, and 1,365 bug-fix pairs—can help drive research in this area.

2022

1. ASE 🏆

   CARGO: AI-Guided Dependency Analysis for Migrating Monolithic Applications to Microservices Architecture

   Nitin, Vikram, Asthana, Shubhi, Ray, Baishakhi, and Krishna, Rahul

   🏆 Distinguished Paper Award 🏆

   In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE ’22), October 10–14, 2022, Ann Arbor, MI, USA. 2022

   Abs HTML PDF

   Microservices Architecture (MSA) has become a de-facto standard for designing cloud-native enterprise applications due to its efficient infrastructure setup, service availability, elastic scalability, dependability, and better security. Existing (monolithic) systems must be decomposed into microservices to harness these characteristics. Since manual decomposition of large scale applications can be laborious and error-prone, AI-based systems to detect decomposition strategies are gaining popularity. However, the usefulness of these approaches is limited by the expressiveness of the program representation and their inability to model the application’s dependency on critical external resources such as databases. Consequently, partitioning recommendations offered by current tools result in architectures that result in (a) distributed monoliths, and/or (b) force the use of (often criticized) distributed transactions. This work attempts to overcome these challenges by introducing CARGO(short for [C]ontext-sensitive l[A]bel p[R]opa[G]ati[O]n)-a novel un-/semi-supervised partition refinement technique that uses a context- and flow-sensitive system dependency graph of the monolithic application to refine and thereby enrich the partitioning quality of the current state-of-the-art algorithms. CARGO was used to augment four state-of-the-art microservice partitioning techniques that were applied on five Java EE applications (including one industrial scale proprietary project). Experiments demostrate that CARGO can improve the partition quality of all modern microservice partitioning techniques. Further, CARGO substantially reduces distributed transactions and a real-world performance evaluation of a benchmark application (deployed under varying loads) shows that CARGO also lowers the overall the latency of the deployed microservice application by 11% and increases throughput by 120% on average.

2. MSR

   Methods for Stabilizing Models Across Large Samples of Projects (with case studies on Predicting Defect and Project Health)

   Majumder, Suvodeep, Xia, Tianpei,  Krishna, Rahul, and Menzies, Tim

   In IEEE/ACM 19th International Conference on Mining Software Repositories (MSR ‘22) 2022

   Abs HTML PDF

   Despite decades of research, Software Engineering (SE) lacks widely accepted models (that offer precise quantitative stable predictions) about what factors most influence software quality. This paper provides a promising result showing such stable models can be generated using a new transfer learning framework called “STABILIZER”. Given a tree of recursively clustered projects (using project meta-data), STABILIZER promotes a model upwards if it performs best in the lower clusters (stopping when the promoted model performs worse than the models seen at a lower level). The number of models found by STABILIZER is minimal: one for defect prediction (756 projects) and less than a dozen for project health (1628 projects). Hence, via STABILIZER, it is possible to find a few projects which can be used for transfer learning and make conclusions that hold across hundreds of projects at a time. Further, the models produced in this manner offer predictions that perform as well or better than the prior state-of-the-art. To the best of our knowledge, STABILIZER is order of magnitude faster than the prior state-of-the-art transfer learners which seek to find conclusion stability, and these case studies are the largest demonstration of the generalizability of quantitative predictions of project quality yet reported in the SE literature. In order to support open science, all our scripts and data are online at \urlhttps://github.com/Anonymous633671/STABILIZER.

3. EuroSys

   Unicorn: Reasoning about Configurable System Performance through the lens of Causality

   Iqbal, Md Shahriar,  Krishna, Rahul, Javidian, Mohammad Ali, Ray, Baishakhi, and Jamshidi, Pooyan

   In The European Conference on Computer Systems (EuroSys). 2022

   Abs HTML PDF

   Modern computer systems are highly configurable, with the variability space sometimes larger than the number of atoms in the universe. Understanding and reasoning about the performance behavior of highly configurable systems, due to a vast variability space, is challenging. State-of-the-art methods for performance modeling and analyses rely on predictive machine learning models, therefore, they become (i) unreliable in unseen environments (e.g., different hardware, workloads), and (ii) produce incorrect explanations. To this end, we propose a new method, called Unicorn, which (a) captures intricate interactions between configuration options across the software-hardware stack and (b) describes how such interactions impact performance variations via causal inference. We evaluated Unicorn on six highly configurable systems, including three on-device machine learning systems, a video encoder, a database management system, and a data analytics pipeline. The experimental results indicate that Unicorn outperforms state-of-the-art performance optimization and debugging methods. Furthermore, unlike the existing methods, the learned causal performance models reliably predict performance for new environments.

2021

1. TSE

   Deep learning based vulnerability detection: Are we there yet?

   Chakraborty, Saikat,  Krishna, Rahul, Ding, Yangruibo, and Ray, Baishakhi

   🏆 Best Paper Award Runner Up 🏆

   IEEE Transactions on Software Engineering. 2021

   Abs HTML PDF

   Automated detection of software vulnerabilities is a fundamental problem in software security. Existing program analysis techniques either suffer from high false positives or false negatives. Recent progress in Deep Learning (DL) has resulted in a surge of interest in applying DL for automated vulnerability detection. Several recent studies have demonstrated promising results achieving an accuracy of up to 95% at detecting vulnerabilities. In this paper, we ask, "how well do the state-of-the-art DL-based techniques perform in a real-world vulnerability prediction scenario?". To our surprise, we find that their performance drops by more than 50%. A systematic investigation of what causes such precipitous performance drop reveals that existing DL-based vulnerability prediction approaches suffer from challenges with the training data (e.g., data duplication, unrealistic distribution of vulnerable classes, etc.) and with the model choices (e.g., simple token-based models). As a result, these approaches often do not learn features related to the actual cause of the vulnerabilities. Instead, they learn unrelated artifacts from the dataset (e.g., specific variable/function names, etc.). Leveraging these empirical findings, we demonstrate how a more principled approach to data collection and model design, based on realistic settings of vulnerability prediction, can lead to better solutions. The resulting tools perform significantly better than the studied baseline: up to 33.57% boost in precision and 128.38% boost in recall compared to the best performing model in the literature. Overall, this paper elucidates existing DL-based vulnerability prediction systems’ potential issues and draws a roadmap for future DL-based vulnerability prediction research. In that spirit, we make available all the artifacts supporting our results: https://github.com/VulDetProject/ReVeal

2. FSE

   Mono2Micro: A practical and effective tool for decomposing monolithic Java applications to microservices

   Kalia, Anup K, Xiao, Jin,  Krishna, Rahul, Sinha, Saurabh, Vukovic, Maja, and Banerjee, Debasish

   In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). 2021

   Abs HTML PDF

   In migrating production workloads to cloud, enterprises often face the daunting task of evolving monolithic applications toward a microservice architecture. At IBM, we developed a tool called Mono2Micro to assist with this challenging task. Mono2Micro performs spatio-temporal decomposition, leveraging well-defined business use cases and runtime call relations to create functionally cohesive partitioning of application classes. Our preliminary evaluation of Mono2Micro showed promising results. How well does Mono2Micro perform against other decomposition techniques, and how do practitioners perceive the tool? This paper describes the technical foundations of Mono2Micro and presents results to answer these two questions. To answer the first question, we evaluated Mono2Micro against four existing techniques on a set of open-source and proprietary Java applications and using different metrics to assess the quality of decomposition and tool’s efficiency. Our results show that Mono2Micro significantly outperforms state-of-the-art baselines in specific metrics well-defined for the problem domain. To answer the second question, we conducted a survey of twenty-one practitioners in various industry roles who have used Mono2Micro. This study highlights several benefits of the tool, interesting practitioner perceptions, and scope for further improvements. Overall, these results show that Mono2Micro can provide a valuable aid to practitioners in creating functionally cohesive and explainable microservice decompositions.

3. arXiv

   Partitioning Cloud-based Microservices (via Deep Learning)

   Yedida, Rahul,  Krishna, Rahul, Kalia, Anup, Menzies, Tim, Xiao, Jin, and Vukovic, Maja

   arXiv preprint. 2021

   Abs HTML PDF

   Cloud-based software has many advantages. When services are divided into many independent components, they are easier to update. Also, during peak demand, it is easier to scale cloud services (just hire more CPUs). Hence, many organizations are partitioning their monolithic enterprise applications into cloud-based microservices. Recently there has been much work using machine learning to simplify this partitioning task. Despite much research, no single partitioning method can be recommended as generally useful. More specifically, those prior solutions are "brittle”; i.e. if they work well for one kind of goal in one dataset, then they can be sub-optimal if applied to many datasets and multiple goals. In order to find a generally useful partitioning method, we propose DEEPLY. This new algorithm extends the CO-GCN deep learning partition generator with (a) a novel loss function and (b) some hyper-parameter optimization. As shown by our experiments, DEEPLY generally outperforms prior work (including CO-GCN, and others) across multiple datasets and goals. To the best of our knowledge, this is the first report in SE of such stable hyper-parameter optimization. To aid reuse of this work, DEEPLY is available here: https://github.com/anonymousplatypus/icse-deeply

4. ASE

   Lessons learned from hyper-parameter tuning for microservice candidate identification

   Yedida, Rahul,  Krishna, Rahul, Kalia, Anup, Menzies, Tim, Xiao, Jin, and Vukovic, Maja

   In The 36th IEEE/ACM International Conference on Automated Software Engineering, Industry Showcase. 2021

   Abs HTML PDF

   When optimizing software for the cloud, monolithic applications need to be partitioned into many smaller *microservices*. While many tools have been proposed for this task, we warn that the evaluation of those approaches has been incomplete; e.g. minimal prior exploration of hyperparameter optimization. Using a set of open source Java EE applications, we show here that (a) such optimization can significantly improve microservice partitioning; and that (b) an open issue for future work is how to find which optimizer works best for different problems.

2020

1. FSE

   MTFuzz: Fuzzing with a Multi-Task Neural Network

   She, Dongdong,  Krishna, Rahul, Yan, Lu, Jana, Suman, and Ray, Baishakhi

   In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ECSE/FSE). 2020

   Abs HTML PDF

   Fuzzing is a widely used technique for detecting software bugs and vulnerabilities. Most popular fuzzers generate new inputs using an evolutionary search to maximize code coverage. Essentially, these fuzzers start with a set of seed inputs, mutate them to generate new inputs, and identify the promising inputs using an evolutionary fitness function for further mutation. Despite their success, evolutionary fuzzers tend to get stuck in long sequences of unproductive mutations. In recent years, machine learning (ML) based mutation strategies have reported promising results. However, the existing ML-based fuzzers are limited by the lack of quality and diversity of the training data. As the input space of the target programs is high dimensional and sparse, it is prohibitively expensive to collect many diverse samples demonstrating successful and unsuccessful mutations to train the model. In this paper, we address these issues by using a Multi-Task Neural Network that can learn a compact embedding of the input space based on diverse training samples for multiple related tasks (i.e., predicting for different types of coverage). The compact embedding can guide the mutation process by focusing most of the mutations on the parts of the embedding where the gradient is high. MTFuzz uncovers 11 previously unseen bugs and achieves an average of 2× more edge coverage compared with 5 state-of-the-art fuzzer on 10 real-world programs.

2. TSE

   ConEx: Efficient exploration of big-data system configurations for better performance

   Krishna, Rahul, Tang, Chong, Sullivan, Kevin, and Ray, Baishakhi

   IEEE Transactions on Software Engineering 2020

   Abs HTML PDF

   Configuration space complexity makes the big-data software systems hard to configure well. Consider Hadoop, with over nine hundred parameters, developers often just use the default configurations provided with Hadoop distributions. The opportunity costs in lost performance are significant. Popular learning-based approaches to auto-tune software does not scale well for big-data systems because of the high cost of collecting training data. We present a new method based on a combination of Evolutionary Markov Chain Monte Carlo (EMCMC) sampling and cost reduction techniques to cost-effectively find better-performing configurations for big data systems. For cost reduction, we developed and experimentally tested and validated two approaches: using scaled-up big data jobs as proxies for the objective function for larger jobs and using a dynamic job similarity measure to infer that results obtained for one kind of big data problem will work well for similar problems. Our experimental results suggest that our approach promises to significantly improve the performance of big data systems and that it outperforms competing approaches based on random sampling, basic genetic algorithms (GA), and predictive model learning. Our experimental results support the conclusion that our approach has bly demonstrated potential to significantly and cost-effectively improve the performance of big data systems.

3. TSE

   Whence to learn? transferring knowledge in configurable systems using beetle

   Krishna, Rahul, Nair, Vivek, Jamshidi, Pooyan, and Menzies, Tim

   IEEE Transactions on Software Engineering 2020

   Abs HTML PDF

   As software systems grow in complexity and the space of possible configurations increases exponentially, finding the near-optimal configuration of a software system becomes challenging. Recent approaches address this challenge by learning performance models based on a sample set of configurations. However, collecting enough sample configurations can be very expensive since each such sample requires configuring, compiling, and executing the entire system using a complex test suite. When learning on new data is too expensive, it is possible to use \textitTransfer Learning to "transfer" old lessons to the new context. Traditional transfer learning has a number of challenges, specifically, (a) learning from excessive data takes excessive time, and (b) the performance of the models built via transfer can deteriorate as a result of learning from a poor source. To resolve these problems, we propose a novel transfer learning framework called BEETLE, which is a "bellwether"-based transfer learner that focuses on identifying and learning from the most relevant source from amongst the old data. This paper evaluates BEETLE with 57 different software configuration problems based on five software systems (a video encoder, an SAT solver, a SQL database, a high-performance C-compiler, and a streaming data analytics tool). In each of these cases, BEETLE found configurations that are as good as or better than those found by other state-of-the-art transfer learners while requiring only a fraction (1/7th) of the measurements needed by those other methods. Based on these results, we say that BEETLE is a new high-water mark in optimally configuring software.

4. NeurIPS

   CADET: A systematic method for debugging misconfigurations using counterfactual reasoning

   Krishna, Rahul, Shahriar Iqbal, Md, Javidian, Mohammad Ali, Ray, Baishakhi, and Jamshidi, Pooyan

   NeurIPS 2020 (Workshop on ML for Systems). 2020

   Abs HTML PDF

   Modern computing platforms are highly-configurable with thousands of interacting configurations. However, configuring these systems is challenging. Erroneous configurations can cause unexpected non-functional faults. This paper proposes CADET (short for Causal Debugging Toolkit) that enables users to identify, explain, and fix the root cause of non-functional faults early and in a principled fashion. CADET builds a causal model by observing the performance of the system under different configurations. Then, it uses casual path extraction followed by counterfactual reasoning over the causal model to: (a) identify the root causes of non-functional faults, (b) estimate the effects of various configurable parameters on the performance objective(s), and (c) prescribe candidate repairs to the relevant configuration options to fix the non-functional fault. We evaluated CADET on 5 highly-configurable systems deployed on 3 NVIDIA Jetson systems-on-chip. We compare CADET with state-of-the-art configuration optimization and ML-based debugging approaches. The experimental results indicate that CADET can find effective repairs for faults in multiple non-functional properties with (at most) 17% more accuracy, 28% higher gain, and 40× speed-up than other ML-based performance debugging methods. Compared to multi-objective optimization approaches, CADET can find fixes (at most) 9× faster with comparable or better performance gain. Our case study of non-functional faults reported in NVIDIA’s forum show that CADET can find 14 better repairs than the experts’ advice in less than 30 minutes.

5. EMSE

   Learning actionable analytics from multiple software projects

   Krishna, Rahul, and Menzies, Tim

   Empirical Software Engineering 2020

   HTML PDF

2019

1. ICSE 🏆

   iSENSE: Completion-aware crowdtesting management

   Wang, Junjie, Yang, Ye,  Krishna, Rahul, Menzies, Tim, and Wang, Qing

   In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) 2019

   HTML PDF
2. PhD Thesis

   Learning Actionable Analytics in Software Engineering.

   Krishna Prasad, Rahul, and others,

   PhD dissertation, Dept. Computer Science, NC State Univ. 2019

   Abs HTML PDF

   Software analytics is routinely used by researchers and industrial practitioners for many diverse tasks. Large organizations such as Microsoft routinely make practice data-driven policy development where organizational policies are learned from an extensive analysis of large datasets. However, despite these successes, there exist some limitations to modern software analytic tools: (1) Lack of relevant data to perform the analytics, and (2) Lack of insightful analytics. This thesis attempts to highlight and offer potential solutions to these pressing problems.

2018

1. TSE

   Bellwethers: A Baseline Method For Transfer Learning

   Krishna, Rahul, and Menzies, Tim

   IEEE Transactions on Software Engineering 2018
2. TSE

   "Sampling" as a Baseline Optimizer for Search-based Software Engineering

   Chen, Jianfeng, Nair, Vivek,  Krishna, Rahul, and Menzies, Tim

   IEEE Transactions on Software Engineering 2018
3. FSE

   Applications of psychological science for actionable analytics

   Chen, Di, Fu, Wei,  Krishna, Rahul, and Menzies, Tim

   In 2018 ACM 26th Symposium on the Foundations of Software Engineering (ECSE/FSE) 2018
4. arXiv

   Hyperparameter optimization for effort estimation

   Xia, Tianpei,  Krishna, Rahul, Chen, Jianfeng, Mathew, George, Shen, Xipeng, and Menzies, Tim

   arXiv preprint arXiv:1805.00336 2018
5. ICSE-SEIP

   We don’t need another hero? The impact of heroes on software development

   Agrawal, Amritanshu, Rahman, Akond,  Krishna, Rahul, Sobran, Alexander, and Menzies, Tim

   In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice 2018
6. ICSE-SEIP

   What is the connection between issues, bugs, and enhancements? Lessons learned from 800+ software projects

   Krishna, Rahul, Agrawal, Amritanshu, Rahman, Akond, Sobran, Alexander, and Menzies, Tim

   In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice 2018

   HTML PDF
7. SWAN

   Characterizing the influence of continuous integration: Empirical results from 250+ open source and proprietary projects

   Rahman, Akond, Agrawal, Amritanshu,  Krishna, Rahul, and Sobran, Alexander

   In Proceedings of the 4th ACM SIGSOFT International Workshop on Software Analytics 2018

   HTML PDF

2017

1. IST

   Less is more: Minimizing code reorganization using XTREE

   Krishna, Rahul, Menzies, Tim, and Layman, Lucas

   Information and Software Technology 2017

2016

1. ASE

   Too much automation? The bellwether effect and its implications for transfer learning

   Krishna, Rahul, Menzies, Tim, and Fu, Wei

   In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering 2016
2. ICSE-BigDSE

   The ’BigSE’ Project: Lessons Learned from Validating Industrial Text Mining

   Krishna, Rahul, Yu, Zhe, Agrawal, Amritanshu, Dominguez, Manuel, and Wolf, David

   In Big Data Software Engineering (BIGDSE), 2016 IEEE/ACM 2nd International Workshop on 2016

2015

1. EAAI

   Entropy based Binary Particle Swarm Optimization and classification for ear detection

   Ganesh, Madan Ravi,  Krishna, Rahul, Manikantan, K, and Ramachandran, S

   Engineering Applications of Artificial Intelligence 2015
2. ASE-Workshop

   Actionable=Cluster+Contrast?

   Krishna, Rahul, and Menzies, Tim

   In Automated Software Engineering Workshop (ASEW), 2015 30th IEEE/ACM International Conference on 2015