With the widespread application of micro-optics in a large range of areas, versatile high quality fabrication methods for diffractive optical elements (DOEs) have always been desired by both the research community and by industry. Traditionally, multi-level DOEs are fabricated by a repetitive combination of photolithography and reactive-ion etching (RIE). The optical phase accuracy and micro-surface quality are severely affected by various etching artifacts, e.g., RIE lag, aspect ratio dependent etching rates, and etching artifacts in the RIE steps. Here we propose an alternative way to fabricate DOEs by additively growing multi-level microstructures onto the substrate. Depth accuracy, surface roughness, uniformity and smoothness are easily controlled to high accuracy by a combination of deposition and lift-off, rather than etching. Uniform depths can be realized for both micrometer and millimeter scale features that are simultaneously present in the designs. The grown media can either be used directly as a reflective DOE, or as a master stamp for nanoimprinting refractive designs. We demonstrate the effectiveness of the fabrication methods with representative reflective and transmissive DOEs for imaging and display applications.