Do I stay with my unbelieving spouse?

I recently came to faith, but my long-term partner is unbelieving. I want to get formally married (we have a common-law marriage), but he does not want to do so. He also objects to me getting baptized. How can I progress in my faith while in this relationship?

The Bible is very clear on how someone in your situation should proceed. Paul says this:

1Cor. 7:13 And a woman who has an unbelieving husband, and he consents to live with her, she must not send her husband away.
1Cor. 7:14 For the unbelieving husband is sanctified through his wife, and the unbelieving wife is sanctified through her believing husband; for otherwise your children are unclean, but now they are holy.
1Cor. 7:15 Yet if the unbelieving one leaves, let him leave; the brother or the sister is not under bondage in such cases, but God has called us to peace.
1Cor. 7:16 For how do you know, O wife, whether you will save your husband? Or how do you know, O husband, whether you will save your wife?

Since you are believing and your common-law husband is unbelieving, you fit the situation Paul describes perfectly. And Paul says to you that as long as your husband consents to live with you, you must remain faithful to him and not send him away or leave him.

On the other hand, you must also pursue your relationship with Christ without hesitation, and regardless of how he responds. You must get baptized, you must study the Bible, you must attend a Bible-based church regularly, etc, because these things are your faithful response to the Lord Who has saved you. As Paul says:

Rom. 12:1 Therefore I urge you, brethren, by the mercies of God, to present your bodies a living and holy sacrifice, acceptable to God, which is your spiritual service of worship.
Rom. 12:2 And do not be conformed to this world, but be transformed by the renewing of your mind, so that you may prove what the will of God is, that which is good and acceptable and perfect.

If your husband determines that your pursuit of Christ is intolerable to him and he decides to leave you, then Paul says you must let him go. Although you desire his affection and his place as father in your family, the word of God says it would be better for you and the children to let him go, than to entice him to stay at the expense of your obedience to Christ. Your obedience to Christ is more important than fellowship with your husband. Moreover, your obedience to Christ is also the most important witness you can give to your children, and it has eternal consequences for you and them.

So remain faithful to your husband in the hope the Lord may use your testimony to bring him to faith also, but if he decides to leave you because of your relationship with Christ, let him go so you may remain faithful to Jesus.