The incident happened around 12:45 p.m. Sunday on Columbia Street near Beech.
Three women in hijabs were walking under scaffolding when a man approached from the other direction. Despite one woman moving to give him room, witnesses say 50-year-old Kyle Allen lunged his shoulder into one of them.
"Fists started flying, and it was his fists. He started swinging at the women, trying to hit the woman that he almost knocked down, but he ended up hitting one of the other women," said a witness named Amy, who called 911 during the altercation.
A hijab is a headscarf worn by Muslim women.
Amy, who asked us not to use her last name, said she also saw Allen rip off one of the women's hijabs from her face.
"As I got closer I heard him calling her names and telling them to go back to where they came from," Amy said.
Cell phone video shot after the incident showed Allen, going back and forth with bystanders. He ultimately walked to his high rise apartment around the corner. Witnesses followed him and pointed police in his direction.
A San Diego Police spokesman told 10News that Allen greeted officers at his unit with a handgun that had a silencer. He was arrested without incident on firearms, battery, and criminal threat charges.
Calls and texts to Allen's phone were not immediately returned. Two police officers were staged on his floor inside his building, but he was not there.
Allen is scheduled to be arraigned Oct. 21.