How Flaky Tests in Continuous Integration


  • About 1.5% test are flaky, which is inevitable. 
  • CI system must be able to deal with certain level of flakiness. 
  • Retrier is good way to solve. and check the histroy of stacktrace,  check if the stacktrace is same as before. 


















MinRanl Concept: the number of layers from one node to another node 

MinRank 3 -5 has highest probablity of flaky 


Some source type may create more flaky test. 




1 person modify the code, unlikely break the code. 2 have best results, if more than 2, the break increase a lot....





Comments

Post a Comment

Popular posts from this blog

Such a cold summer

I almost did not wear the t-shirt alone this summer, always have a jacket on when I went outside. Most of time the temperature is lower than 20 c-degree. Totally did not get the feeling of summer. What a pity.
Read more

My Unsolve Questions

How to reduce the admin work for tester? test Measurement, Management, Communication, Automation.  Agile: Test Might take longer time than development.  How to solve that in Agile? how to make exp test efficient in scurm. does tester be scurm master better sit with developer or tester in scrum should we can Quality advisor Automation test negitive test?
Read more

My interview questions to a company using SAFe.

What is the setup of your QA team?  What is the ratio of Test vs Developer Do you have the system layer test?  Do you have the non function test layer?  How do you divide the train, do you have plat form team, and do they have their own train? Any support work in the team` What is the good part, and what is the bad part How long does the team learn how to do PI planning.  What is the biggest challenge you meet,  Did you have any challenge when you split up the team?  Any suggestion on how to devide the team, and what is the practise in lego.  In the new setup, is it possible different team work on a same product of same feature?  BDD:  In which level do you apply the BDD, how do you use it in integration level? who responsible for design and implemented BDD? Agile: What is your product, do you have technical(platform) team, how do you make them work in the agile framework.  Special list can swap among ...
Read more