Anything that distracts man from the remembrance of Allah is haram. This is precisely why the Prophet said that everything in this world- except good deeds and prayer- is cursed.Yet, masturbation is forbidden (or at least discouraged). Go figure.
However, not everyone was destined to be a prophet, and not everyone has the mental capacity to be a saint. Allah is only concerned with the content of a man’s heart, and humans were manufactured to sin. In other words, a man who lived an entirely sinful life may end up in heaven, while someone who did his best to avoid all worldly pleasures may not!
The majority of people don’t know what the hell is going on anyway, which explains why the Prophet said that heaven is mostly filled with naive and clueless people.