the Map my ride page isn’t loading but, if you thinking of riding out to Woods Point from lilydale, then thru to Walhalla and back to Lilydale then your right in that it is doable on a CX bike, the roads are ok from last time I was there but do get corrugated, also yes there is a lot of climbing.
Me an some mates did Warby-Jamison via Big River road, then Jamison-Erica then Erica to Moe via Baw Baw in August last year. The roads around Baw Baw are very steep, although the south face road (gravel from Erica) is really nice.
For an alternative I would suggest:
Day one- ride to a campsite on Big river road (via reefton spur) from Warby or Lilydale, there are plenty of campsites lots will be full of boguns some should be free, it’ll be a tough ~100kms but not ridiculous.
Day Two - Finish riding Big River road* and ride to Eildon for lunch/afternoon beer, then head west to a campsite on Murrindindi Scenic reserve (or at the northern end of the Cathedral ranges if there are spots)
*the northern half of Big River road gets pretty rough -4wd- but its a great road, great views, swim in the river if its hot, easy to navigate.
Day Three- Head south along Murrindindi road to Toolangi and then out to Hurstbridge
That works out at less than 100kms a day still heaps of gravel and climbing but a bit less hectic.
Woods point is great but you kinda have to do 150km big days to fit a good loop into 3 days… plus the pub is booked out.
Also suggest the Gippy Gold Loop with (or without) some extras (like a commute from Bunyip)
Third suggestion which I have ridden that is good is:
1: Train to Marshall/end of the Geelong line, ride into the Otways, camp somewhere in there
2: pickup Mnt Sabine Rd and follow to Forrest
3: ride dirt roads to Gellibrand to pickup the rail trail to Colac.
Its the shortest of the three options (more like ~70kms day maybe) but can extend with side trips down off the ‘spine’ of the otways down to the great ocean road and such.