Anybody who has sat through a particularly contentious synagogue board meeting knows that Jews still practice the ancient custom of trial by ordeal. But this is about the only example still surviving. The Rabbis prefer judicial methods that involve evidence, witness testimony, and the examination of the word of law over the examination of divine responses in determining who and what is right (Baba Metziah 59b).
Yet before the Talmudic period, trial by ordeal popped up, both in the sacred past, but also in at least one legal ritual. An ordeal is a ritualized test that one or both litigants must undergo to be vindicated. We are most familiar with the concept through "ordeal by combat": let the two litigants hack it out, and the gods (or God) will ensure the righteous party will win. Yeah, I bet that concept was just as suspect to the participants as it is to you and me. Napoleon got it right when he said, "God sides with the big battalions." I also know you are thinking about David and Goliath. Indeed, David may have framed that fight as a test of truth and right, but Goliath and I would agree it was really a championship bout for a symbolic victory, just as such contests were in Homer. I wonder how many people assumed these one-on-one contests established the right or wrong of the fight (maybe the winners did).
No, ordeals in the Bible involve inviting God to reveal the truth of the matter through a non-violent test. Examples include determining who should have leadership, such as the face off of Moses vs. Korah (Num. 16), or the dueling staves between Aaronites vs. the other tribes (Num. 17:16-26). Ordeal is also used to select who was worthy to fight for God in a milchamah mitzvah - a divinely ordained war (Judges 7).
Still, none of these are actually formal judicial proceedings. These were one-time crisis tests. The only example of a true legal ordeal appears, not surprisingly, in the book of Numbers, where most of the ordeal stories are recorded. In Numbers 5 we read about the Sotah. In this trial, a wife suspected of infidelity (it may be she had to be pregnant, the text is unclear), she is required to drink a potion made up of earth from the Temple grounds combined with a written curse - possibly written in the earth itself - that has been dissolved in living water. If she is guilty, she (or the fetus) will die and so, as it were, "return to dust." If innocent, the power of the potion would make her pregnant, or, alternatively, the resulting child from the existing pregnancy will "be like Abraham" (Mishna Sotah 2:2; Sotah 17a). This link to Abraham may be because of his righteousness, because God made him endure ten 'tests,' and/or because he once trod on the Temple grounds, infusing the earth with his holiness.
The ordeal of the Sotah was abolished from Jewish judicial practice very early in the Talmudic period, but became the model for magical potions and formulae in Jewish and Christian magic, which often require the adept to write a divine name or incantation on an edible object or dissolves the words in water and drink them.