Help! Why is my 14 week old only drinking 2oz every 3/4 hours?

I went to the health visitor today she said there is nothing to worry about, he only gaines 4ounces in 2 weeks, and has been really off with his feedings so she told me to set him a time frame of feedings and feed him every 4 hours... she told me not to worry as his weight is still above average but i cant help but tear my hair out when he refuses the rest of his bottle :(