In the main menu it says metres and Kilonewtons where as in the general tab for water mass density not sure what to apply either kg/m3 or kN/m3 ? If we apply in kN/m3 the wave output shows 10.2 kg/m3 which is not right.

It is the water mass density – specified in kg/m3. So, the value should be 1000 kg/m3. Note that it is not the water weight density. So, it does not matter if the force system is set to KN.