A compact and highly stable high pulse energy passively mode-locked fiber laser is presented. A segment of Yb3+-doped double-clad large-mode-area fiber with extremely low nonlinearity is developed as the active medium. The self-starting mode-locked operation is achieved by the cooperation of nonlinear polarization evolution and semiconductor saturable-absorber mirror. The fiber laser generates laser pulses with the average power of 160mW at the repetition rate of 55.9MHz (corresponding to pulse energy of 3nJ), and the pulse duration is 10.6ps.