I think it varies on what you are looking for.
Private schools can offer opportunities not available at public schools. Examples can be preparatory academies, or a more major focused school. Regardless though, they MUST teach general educations that all students should learn from (Mathematics, Science, English, etc...).
Public schools can vary often, which is frankly saddening. My school I go to is fantastic. Brand new equipment, 9/10 of the teachers are great, half descent staff. Hell, GreatSchools.org grants us a 10/10 (mainly based on test scores). The issue is that the way public schools are funded put those living in poorer areas more at risk. That is one of the flaws in the US education system. Those in wealthier locations have better schooling, hence they are more likely to make more money.
Now in terms of private schools for religious reasons I am more on the fence. I will straight forward admit I'm a bit biased as I attended a Christian Middle School, and I have to say the education was very different from what my current friends experienced. We had less Extracurriculars (once again a cost issue, based on where you live) meaning I didn't have the chance to learn an instrument. The science class was also different. Because science was molded more around the religion some details were either left out or molded. I can understand if someone decides to not accept/believe it, but teaching it is another matter at hand. Although, I will say as long as the general education classes are fine I'm okay with these schools.
One thing I find fascinating is the perception that private schools are instantly better, when the best teachers I've ever had were at a public school. But once again, the problem lies in location and money