Mount Fuji Climb in Off Season Yoshida Trail

To reach Mt.Fuji 5th Station (Yoshida trail), you have to catch two buses :

Gotemba station to Mt.Fuji Station: 2540 Yen for a Roundtrip (65mn)
Mt.Fuji Station to Mt.Fuji 5th Station: 2100 Yen for Roundtrip (62mn)
The total duration from Tokyo to Mt.Fuji 5 Station is around 4 hours.
If you plan to make a one day climb, I recommend you to stay somewhere around Gotemba Station so you can catch the earlier bus to the 5th Station and you have a lot of chance to climb to the summit and back to the 5th station to catch the last bus at 17h45.
It is depending on your physical condition.