The teenage migrants were trying to reach the Gulf countries via Yemen on Wednesday.
In a statement
, the International Organization for Migration described the drownings in the Arabian Sea as "shocking and inhumane."
"The survivors told our colleagues on the beach that the smuggler pushed them to the sea when he saw some 'authority types' near the coast," said Laurent de Boeck, the Yemen mission chief for the International Organization for Migration.
"They also told us that the smuggler already has returned to Somalia to continue his business and pick up more migrants to bring to Yemen on the same route."
Shortly after the drowning, IOM staff found shallow graves of 29 migrants on a beach in Shabwa, the UN agency said.
"The dead had been quickly buried by those who survived the smuggler's deadly actions."
The migrants were escaping the political instability and violence plaguing their nations. The IOM estimates that since January, 55,000 migrants have left the Horn of Africa to come to Yemen. Smugglers work in the Red Sea and the Gulf of Aden.
More than 30,000 of those migrants are under age 18 and from Somalia or Ethiopia, while a third are estimated to be female.
The trip across the waters from Somalia to Yemen is fraught with peril, with Yemen engulfed in war. A boat carrying Somali migrants
was attacked in March, killing 42. Windy season in the Indian Ocean makes the trip especially dangerous.
The smuggler's boat approaching Yemen on Wednesday was near the Shabwa coast. The IOM said the smuggler pushed 120 Ethiopians and Somalis into the sea.
Medics provided care to dozens of survivors, but some left the beach before help arrived. Twenty-two people are missing.
The average age of the people headed to Yemen on Wednesday was 16.