The firm's founder has put up an award of $200,000 (£122,000) worth of Bitcoins for breaking the protocol.
To win the award, users will have to intercept a daily message between the founders Pavel and Nikolai Durov to find out a secret email address.
They have to send the decrypted text of the message to the email address to prove they have cracked the protocol.
Telegram, which says it is "the fastest and most secure messaging system in the world", said users would also have to give a detailed description of how they cracked the encryption.
The firm says it will publish encrypted Telegram traffic between the two founders on its website and users can view it in real time.
"To prove that the competition was fair, we will publish the participating keys necessary to decrypt the traffic as soon as a winner is announced," the firm said.
In case there is no winner by 1 March 2014, encryption keys will be published on that date.
Source: BBC News
