Proper initialization of the nonlinear optimization is important to avoid local minima in phase diversity wavefront sensing (PDWS). An effective neural network based on low-frequency coefficients in the Fourier domain has proved effective to determine a better estimate of the unknown aberrations. However, the network relies significantly on the training settings, such as imaging object and optical system parameters, resulting in a weak generalization ability. Here we propose a generalized Fourier-based PDWS method by combining an object-independent network with a system-independent image processing procedure. We demonstrate that a network trained with a specific setting can be applied to any image regardless of the actual settings. Experimental results show that a network trained with one setting can be applied to images with four other settings. For 1000 aberrations with RMS wavefront errors bounded within [0.2 λ, 0.4 λ], the mean RMS residual errors are 0.032 λ, 0.039 λ, 0.035 λ, and 0.037 λ, respectively, and 98.9% of the RMS residual errors are less than 0.05 λ.