That is not exactly how Valium (diazepam) works.
Tolerance is not just based upon previous exposure it is also based on genetics. It is actually a lot like alcohol, some people are drunk after a single drink and others need 10 drinks to feel it. And the fact you are male and a big person (and I'm assuming young) probably means you have a naturally high tolerance. Also people of European decent typically have a higher tolerance and more rapid metabolism of Valium, other benzodiazepines, and especially alcohol.
Valium works in 20-30 min, it actually has the most rapid onset of action of oral benzodiazepines aside from a few hypnotics.
Also if you are taking it for medical reasons (rather than to get high) just because you don't feel anything does not mean it is not working. Often people notice their anxiety is better due they don't feel "high," sedated, or anything else. Also if you end up taking it continuously and long term no matter what dose you take you will develop selective tolerance to the hypnotic, anticonvulsant, and euphoric effects. However people DO NOT develop tolerance to the anxiolytic effects.
If you are taking Valium for medical use you may need to give it a few days for it to help or you may need a higher dose. In many cases when a person takes a dose that has no apparent effects after a few days it will start to help. Generally 10 mg of Valium is the maximum single dose and the typical maximum dose is 40 mg/day. In some cases people need a more potent drug like Ativan (lorazepam) or Xanax (alprazolam).
More information about Valium:
US Prescribing information:
http://www.gene.com/gene/products/information/
Canadian Prescribing Information:
http://www.rochecanada.com/portal/eipf/ca/portal/roche/consumer_information?paf_gear_id=17700009&paf_pageId=re7191019&glossary_id=static/glossary/re7300002/re77300002/re77300003/re753001/Definition_01054.content