A modulational instability analysis is carried out for a new kind of integro-differential nonlinear Schrodinger equation which describes nonlinear wave-envelope propagation for several different situations, such as the transport of very intense charged-particle beams in accelerating machines, large amplitude electromagnetic wavepacket propagation in nonlinear media (optical fibres, plasmas, etc.), Langmuir-wave-envelope propagation in warm plasmas, and collective dynamics in mesoscopic physics. The modulational instability analysis is extended from configuration space into phase space, by using the Wigner transform. It is shown that the propagation of a wavepacket in a nonlinear medium, governed by the above nonlinear Schrodinger equation, ...