Joseph learned that Mary was pregnant, but not by him, so he started to make plans to divorce secretly, not wanting to make her a public example, because he cared for her. However, an angel appeared to Joseph in a dream and explained the situation to him,
“Then Joseph, being aroused from sleep, did as the angel of the Lord commanded him and took to him his wife, and did not know her till she had brought forth her firstborn Son. And he called His name Jesus” (Matthew 1:24–25).
Joseph always wanted to do the right thing, and since Mary had not been with another man, Joseph took her back, and did not have sexual relations with her, because the angel had told Joseph that the Lord was fulfilling Isaiah 7:14 in Mary, that a virgin would conceive and bear a son.
Therefore, it must have been known publicly in some way that he had not been with her in that way until after the birth of Jesus.