Livestock, agricultural soils, crop burning, deforestation, cropland, grassland are all used by livestock for dairy and meat. Non-dairy food sources does have similar uses, but they are a lot more efficient calorie or protein wise.
I'm not sure about how much manure is needed to sustain mostly plant-based agriculture, but it sure is a lot less than a meat-heavy one. Would love data about this.
I don't care much about being right, I care about being honest about my own impact.