If you have connected with someone either via conventional family tree research or through autosomal DNA SNP testing and both you and your connection want to conduct DNA testing to investigate the relationship in more detail, a good choice would be Y-STR or mtDNA testing if you think you may both belong to a branch of the same unbroken male or female line.
If you both have a direct line to a recent male ancestor then, in many cultures, it is likely you will also share the same surname (the Y chromosome is inherited in the same way as surnames though the male line). This might not always hold true of course because your surname may have changed through an adoption somewhere in the past or because at some point in the chain there has been an unrecorded and unrecognized paternity that has disrupted the pedigree.
Unlike autosomal SNP testing, which relies on statistical knowledge of the extent to which DNA shuffles and recombines through the generations and can’t determine relationships absolutely especially when there are several generations between two people, Y-DNA and mtDNA give more of a “yes-no” answer.
If your Y-DNA and mtDNA results are very different there is no chance at all that you are closely related through unbroken male or female line and you do not have a recent common ancestor and so these tests are very good at closing off such lines of enquiry.
If your results match then there is good evidence that you belong to a branch of the same maternal or paternal lineage. The question now is; how far back did your bloodlines cross? As Y-STR and mtDNA are inherited unchanged through generations without any mixing, logically you’d think it would be impossible to say when two people had a common ancestor, in fact, if there were never any changes we’d expect everyone to have the same result and be the same type.
The reason different mtDNA and Y-STR types exist (known as haplotypes) is because of DNA sequence changes that occasionally occur when DNA is copied from one generation to the next. In other words sequence changes that happen during formation of egg and sperm cells. The changes occur due to naturally occuring DNA copying errors, called mutations, when new cells are created. DNA mutations are transferred to the next generation quite rarely and the rate differs depending on factors such as the type of mutation, where it is situated on the chromosome and whether or not it has an impact on the individual’s ability to procreate (mutations can cause a biological malfunction or a physical disorder and can be fatal).
Mutations in the mtDNA and Y-STR DNA that we routinely test are well studied and the rates are well defined. Using mutation rates it is easy to work out how many mutations you’d expect over a certain number of generations for a particular test and so, if two people match exactly, you can give a good estimate of how recently they share a common ancestor. It turns out that mtDNA mutates very slowly and, whilst non-matching results give conclusive proof that you don’t belong to the same maternal line, if you are an exact match it is not possible to say whether your common ancestor lived recently or a long time ago, ie. many generations ago.
Y-STRs mutate more frequently than mtDNA and matches are a little more informative especially if you choose a test that analyzes many Y-STR regions simultaneously. The more Y-STRs you have tested the more likely you will find a mutation. If you don’t find a mutation and still match with someone at many Y-STRs the upshot is it is more likely you share a recent ancestor along your male line. A good illustration is the range of Y-STR tests offered by Family Tree DNA; the cheapest ($49) simultaneously tests 12 Y-STRs, matching individuals are likely to share a common ancestor anywhere within 29 generations – not very useful if your trying to find new family members. The most expensive ($359) tests 111 Y-STRs and people who match are likely to share a common ancestor within 4 generations. Of course, if you are not related, both the 12 Y-STR test and 111 test are very likely to give you conclusive proof that you are from different male lineages because your results will almost always be different.