Summary: Testing with 5 users has become a commandment in UX research. Worse yet, a sample of 5 has become the rule, for any type of user research (field studies or diary studies). Sample size is a big deal in UX because it impacts your learning and decision-making. Skimp on sample size and you will likely cut yourself short of valuable insights.
For Usability Testing: If you follow Jakob Nielsen's advice: test with 5 users or less (2 users for low-fidelity prototypes) and do many iterations of testing (at least 3 rounds of testing). Our advice: if you have multiple user segments and you don't have time to play with 3-5 iterations, test with 8-10 users for prototypes, and 15-20 users for finished products. If you can, or need to iterate then a second round of testing should suffice.