How much memory can a process use?

faberman is right. if you want it to fail during the memset properly you need to check malloc () else memset might fail because you're passing it NULL as a destination address. the loop would look better this way:
while ((waste = malloc(STEP*MB)) != NULL) {
memset(waste,1,STEP*MB);
counter+=STEP;
printf("%i MB wasted\n",counter);
}
That way the loop would terminate if malloc() fails. Because if malloc fails, memset will fail, but it won't fail because of the OOM killer. It will fail because you called it improperly.

chances are the OOM killer will kill on a memset before this happens, but it is entirely possible, so for the code to work right every time, you need to check malloc ()

Also the cast to (char *) is redundant.