How do you figure? According to Wikipedia, "the Eastern Baltic languages split from the Western Baltic ones between AD 400 and AD 600", "The differentiation between Lithuanian and Latvian started after AD 800", "The earliest surviving written Lithuanian text is a translation dating from about 1503–1525", etc.
Lithuanian is closest to the original proto-indo-european language keeping more of it's features than any of the hundreds of other languages that spawned out of it.
It's hard to say any language is "oldest" because languages are alive and the people that speak them change constantly. The pieces that make up the language though are indeed very old regardless of the fact that "Lithuanian" hasn't existed nearly as long.
>Among Indo-European languages, Lithuanian is extraordinarily conservative, retaining many archaic features otherwise found only in ancient languages such as Sanskrit[5] or Ancient Greek. For this reason, it is one of the most important sources in the reconstruction of the Proto-Indo-European language despite its late attestation (with the earliest texts dating only to c. 1500 AD). The phonology and especially the nominal morphology of Lithuanian is almost certainly the most conservative of any living Indo-European language,[4][6] although its verbal morphology is less conservative and may be exceeded by the conservatism of Modern Greek verbs, which maintain a number of archaic features lacking in Lithuanian, such as the synthetic aorist and mediopassive forms.
Related languages are different due to changes; but they haven't all changed in the same way. Thus it is perfectly sound to describe one language as closest to the common substrate of other related languages.
Looking only at languages L1 through L3 we may infer a common ancestor from what is more shared. Saying that L4 is closest to this hypothetical language is not an empty concept, or artificial based on some ideology.