No. Latex allergy is faced by negligible men and reports of mild to severe allergic reactions occur very rarely.
Severe allergic reactions of using latex condoms are extremely rare.
To know whether you are allergic to latex, think whether you had any allergic reaction while using rubber gloves or balloons in the past.
If you had allergic reaction then, chances of your body reacting to latex condoms are high.
In those rare cases try Non Latex Condoms