I tried out using the 19mm collet but the little thin disc fell right through. So in tightened the nut and carefully put in the roller till it started gripping. The problem using this method of holding is, I'm not sure how to hold it such that the working surface is perpendicular to the spindle. Further, I'll have problem with those pieces that have the blue plastic ring on their parameter.
Looks to me that holding the rollers via the centre hole is the way to go.
If anyone has any suggestion, please comment. Thank you.