If I just backup my config stuff in /etc/openvpn will I be fine if something goes wrong?
in theory, yes; I'd also have a look at the ubuntu specific network file (/etc/network ?) to see if there's anything tun/tap/openvpn related there.
I won't have to regenerate keys or certs or anything, right? Once I start up the service again my clients will be able to connect as before?
again, in theory, yes; do you have the master ca.crt and ca.key files? were the file generated in the /etc/openvpn directory as well? or elsewhere on the system?