It can be done without dairy (although I eat butter and ghee - if those don't work, you could easily use lard, tallow, coconut oil, etc.) or nuts, but it is more challenging, especially at higher calorie levels. I eat: (i) very fatty cuts of meat; (ii) add fats to my meats -- I pore the extra fat over my food so I eat all of the calories; and (iii) make a small treat using butter or ghee (I've done coconut oil, but I don't respond well to it), a tad of unsweetened cocoa powder, and stevia. If you can tolerate non-animal sources of fats, you can add avocado, olives, etc.
Of course, this is not necessarily desirable for everyone