How to extract only numbered rows from a column

How to extract only numbered rows from a column

For e.g: I have a column named "Card No", its a varchar but majority of
the rows contain numbers and few contains alphanumeric numbers. I want to
ignore all the records that are not numbers.
Below are a few examples for the same:
Select CardNo from XX;
Result:
CardNo:
_______
1234,
5467,
1234/5467,
abc 667,
efg428
In the above result set, I want to eliminate the last three values. Please
advise.