Iptable ubunto allow multiple specfic ip's to a specific port, drop the rest?

Iptable ubunto allow multiple specfic ip's to a specific port, drop the rest?

Wanting to do something like this
iptables -I INPUT ! -s $IP_1 -p tcp --dport $SERVER_PORT -j DROP
iptables -I INPUT ! -s $IP_2 -p tcp --dport $SERVER_PORT -j DROP
iptables -I INPUT ! -s $IP_3 -p tcp --dport $SERVER_PORT -j DROP
Where the server port is the same for all, but the allowed ip's can be
different.... from what I know of iptables this wouldn't work as IP_3
would be dropped from the first rule... so how do I get something like
this to work?