I have always kept film and paper in the freezer and I'm not particularly careful about it. I will, generally, put sheet film boxes and rolls/35mm into freezer bags, but other times I simply toss the box into a basket. When I want to use it, I remove from the freezer and let it warm in my darkroom sink. I've never had any issues, that I'm aware of, working this way.
I guess storing film in a "cool place" is relative to where one lives. In the summer here, my indoor ambient temps reach 28C with outdoor temps hitting 46 - 49C! I don't feel comfortable leaving expensive sheet film in a drawer.
I agree, though, that fast films will change even if freezer stored.