I grew up chubby as well. I haven't touched milk in years.
What are your goals? Fat loss, weight gain?
For fat loss i would say skip it. It is very insulinogenic, so releasing tons of insulin. Not great for fat loss.
For weight gain, it may be ok for you due to the increases in IGF-1 it causes. But, that effects all tissue, muscle, fat, bone etc.
he talks about a bunch of stuff, but he mentions the increase in IGF-1 due to milk intake.