How can I set up an RF660VPN for SMTP proxy but not be an OPEN SMTP relay? The Last time I did it we were put in the RBL (Realtime black list).

In the SMTP proxy setup page, make sure there are no entries in the "Mail relay for" option.