The speed of population growth and decrease depends on Ministry and Department funding, taxes and social benefits. The lower the taxes, the more stars there are in other Ministries and the higher the salaries and personnel number, the faster the population growth. If there is not enough Ministry funding and the taxes are too high, population can decrease.
Population can also increase as a result of territory expansion, because the population of the conquered territory will be added to your population. Population decreases when you lose territories.