No one was hurt during the scare on the Cobham Aviation jet bound from Perth for Barrow Island in Western Australia, but some passengers panicked.
"Fuel starting spewing out, caught alight, there was a bit of panic on board but the pilots were quite quick to react and cut fuel lines and then put it out," Jason Grimmett, who was on the plane, told the Australian Broadcasting Corporation.
"So we just turned around and came back in."
Grimmett said most of those on board remained calm as the four-engine plane returned to the terminal, but not all.
"There was a couple of guys that were panicking quite bad but we've still got three other engines, so unless something major goes wrong, just a bit of fuel caught alight," he said.
A spokesperson for the government agency Airservices Australia told AFP there was a fire in the number two engine shortly after departure.
"The pilot shut it down, the flames were extinguished and the plane returned to Perth," she said.
"It landed safely and was able to taxi to its gate. The fire services provided an escort on the ground."
Witnesses on the ground said they saw flames billowing from an engine of the British Aerospace 146 jet while it was in the air and feared the worst.
"As soon as it got up over the buildings, I could see a 20m or 30m trail of red and white smoke coming from the left-hand engine," said a caller to local radio station 6PR, identified only as Gary.
"It was climbing at the time and it just looked terrible, it looked like it was going to be a disaster."
The airline is a charter company servicing regional Western Australia.