The Messenger of Allaah (sallallaahu ’alayhi wa sallam) said, “No Prophet from amongst the Prophets will have as many followers as me. Indeed, there will be from amongst the Prophets he who had no one believing in him, except one man.” 
Imaam Muhammad Naasirud-Deen al-Albaanee (d.1420H) commented upon this hadeeth, saying, “In this hadeeth is a clear proof that the abundance of followers, or lack thereof is not the measuring standard that indicates whether the caller is upon truth or falsehood. So even though the da’wah of these Prophets – ’alayhimus-salaatu was-salaam – was one and their Religion was one, their varied with regards to the numbers of their followers. Some of them had many and some of them had a few, to the extent that some had no one to believe in them except one man. Rather, there was from amongst them he who had not a single follower!
So in this is a profound lesson for the callers and those who are being called in this age of ours. So it is obligatory upon the caller to remember this reality. He must remain for a long time in the path of calling to Allaah the Exalted, and he must not pay any attention to the scarcity of those who answer him, because there is nothing upon him, except to convey the message clearly. And he has the best example in the previous Prophets who did not have anyone with them other than one or two men!
So the one being called upon must not despair due to the scarcity of those who answer the caller, and he must not take that as a reason for doubting in the da’wah of truth and thus abandon eemaan (faith) in it, let alone that he would take that as an indication of the falsity of the da’wah with the argument that no one is following this caller, or that only a few are following him. And if his da’wah were truthful, the majority of the people would not follow him! Allaah the Mighty and Majestic said,
“And most of mankind will not believe, even if you desire it eagerly.” [Soorah Yoosuf 12:103].” 
: Related by Muslim (1/130).
: Refer to Silsilatul-Ahaadeethis-Saheehah (no. 397) of al-Albaanee.