Sri Chinmoy: There are two reasons why you lose your joy. One reason is that your mind starts functioning most powerfully, vehemently. While functioning in this way it allows obscure, impure, undivine thoughts to come in either consciously or unconsciously and then its joy has to disappear. But if purity is well established in the mind, then the joy will last for a long time.
Another reason you might lose your joy is because your vessel is small and you have taken Light, which is joy itself, beyond the capacity of your vessel. The quantity of Light that you have gotten during your meditation has satisfied you, but the excess you were not in a position to hold in your vessel; so you felt miserable.