Too much automation? The bellwether effect and its implications for transfer learning

31st International Conference on Automated Software Engineering

Recommended citation: Krishna, R., Menzies, T., & Fu, W. "Too much automation? The Bellwether Effect and its Implications for Transfer Learning." In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE 2016). ACM, New York, NY, USA, 122-131. DOI:


“Transfer learning”: is the process of translating quality predictors learned in one data set to another. Transfer learning has been the subject of much recent research. In practice, that research means changing models all the time as transfer learners continually exchange new models to the current project. This paper offers a very simple bellwether transfer learner. Given N data sets, we find which one produce the best predictions on all the others. This bellwether data set is then used for all subsequent predictions (or, until such time as its predictions start failing– at which point it is wise to seek another bellwether). Bellwethers are interesting since they are very simple to find (just wrap a for-loop around standard data miners). Also, they simplify the task of making general policies in SE since as long as one bellwether remains useful, stable conclusions for N data sets can be achieved just by reasoning over that bellwether. From this, we conclude (1) this bellwether method is a useful (and very simple) transfer learning method; (2) bellwethers are a baseline method against which future transfer learners should be compared; (3) sometimes, when building increasingly complex automatic methods, researchers should pause and compare their supposedly more sophisticated method against simpler alternatives.


 author = {Krishna, Rahul and Menzies, Tim and Fu, Wei},
 title = {Too Much Automation? The Bellwether Effect and Its Implications for Transfer Learning},
 booktitle = {Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering},
 series = {ASE 2016},
 year = {2016},
 isbn = {978-1-4503-3845-5},
 location = {Singapore, Singapore},
 pages = {122--131},
 numpages = {10},
 url = {},
 doi = {10.1145/2970276.2970339},
 acmid = {2970339},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {Data Mining, Defect Prediction, Transfer learning},