It's very dangerous to label an unidentifiable snake as "harmless". We have no way of telling what it is without photos, and baby brown snakes CAN be unbanded. Don't touch until you're absolutely sure what it is, but all baby venomous snakes ARE venomous and can easily put you in hospital.
I'd suggest getting a photo so people can identify it, or call a snake removalist.
Edit: Blind snakes look exactly like worms (except scaled instead of slimy) and aren't capable of moving very fast over ground.