Found a solution - the IIF should be inside the dateadd:
=format(dateadd("s",IIF(Fields!Connected_To_An_Agent.Value="0","0", (Fields!Return_Call_Hold_Time_sec.Value / Fields!Connected_To_An_Agent.Value)),"00:00:00"),"HH:mm:ss")