I have read in Data Communications and Networking, 5th Edition, by Forouzan that the formula for calculating the bandwidth for QAM modulation is \$B=(1+d)S\$ where \$d\$ is factor that depends on filtering process and is between 0 and 1, and \$S\$ is symbol rate.
My question is why is that formula used instead of e.g. Nyquist or Shannon formula for capacity to calculate bandwidth?
Because they are computing different things, used for different purposes.
In the formula you quote, bandwidth means amount of 'occupied bandwidth' in the RF spectrum. This is the amount of channel width that must be dedicated to transmitting the signal. Theoretically, given enough transmission time and computing power, d could approach zero. In practice, these are traded off with d varying between 0.2 and 0.5 for different systems. It's independent of the type of modulation used, whether 4QAM or 256QAM.
Nyquist uses bandwidth with respect to the sampling theorem. In a sense, the factor d is a measure of how far 'above Nyquist' the bandwidth is, so the relationship is quite close.
Shannon has to do with the capacity of a channel when noise is taken into account, which is a different purpose to your formula. Here the different between 4QAM and 256QAM is profound.