The saying goes back to the days of small family farms, when each farm had livestock that grazed on the family land. If your cows or sheep got into your neighbour's meadow through a gap in the fence, your neighbour would have less grass and hay for his own animals. They might also damage your neighbour's crops.
So farmers would make sure that fences (sometimes walls made of loose rocks) divided their property. Otherwise they would have angry disputes and perhaps go to court.
By the way, "neighbour" comes from an old English expression, "nigh boor"—a "near farmer."
It doesn't necessarily mean that you do not have a close relationship. It means that in friendship you need to be careful not to overstep certain boundaries. In other words, you need mutual respect for the friendship to grow and continue.
Robert Frost used this phrase twice in "Mending Wall"