Fire place was insulated with that foam crap - want to remove it?

How hard is it to remove that ugly yellow insulation stuff? It is in the fireplace vents, and they put plywood in the actual fireplace and used the insulation to hold it in.