It is pollution for as long as you use fossil fuels (== carbon trapped underground long ago gets released to atmosphere).
Growing stuff to create hydrocarbons uses carbon (CO2) already there in the atmosphere and releases no new carbon to it, thus it isn't pollution.
Yeah, but when we burn it in cars and heaters and power plants, it gets released again. There is no denying of the negative influence on the atmosphere (unless you're George Bush, that is )