Divorcee and Cohen

For those with any familiarity with Jewish marital laws, it is common knowledge that whereas a Cohen can marry a widow, he cannot marry a divorcee. In the words of this week's parsha: 

"They shall not take [into their household as their wife] a woman defiled by harlotry, nor shall they take one divorced from her husband. For they are holy to their God."

My simple question is why is there something less becoming for a Cohen to marry a divorcee than a widow. Humbly, I would like to admit that I have no good answer. 

A friend, Sami Molcho, answered that his understanding was that it's imperative for a Cohen who brings people closer to Hashem to have clarity of mind, thought and spirit, the equanimity that stems from marrying someone without "baggage," a clean slate. 

A widow knows her husband will never return and gains a certain level of closure - perhaps - on her husband's passing. The relationship terminated with her having been married to him, whereas a divorcee, even if the betrothal was severed and the couple had never actually gotten married (nowadays halachically actionable betrothal and marriage take place simultaneously), was not allowed to marry a Cohen. Perhaps, Sami would argue, the same logic goes for a convert. Or likewise, someone who was engaged and underwent chalitza. 

I am not sure if my friend, Sami, is right, but it most certainly provides food for thought. There is nothing inherently unholy about a Cohen marrying a divorcee, convert, or woman who underwent chalitza, but it would seem - as the verse insinuates - because the man has to have the perspicacity, and clarity to carry out his work without fear, hesitation or uncertainty, the Torah - Sami argues - chose to forbid a Cohen from marrying a woman who could perhaps diminish his ability to do so.