If you can live with security against passive adversaries here is how you could do that: Denote the inputs of Alice and Bob $x$ and $y$ respectively. Bob generates public- and private-key for the FHE scheme. He sends the public key to Alice a long with an encryption of his input. Alice encrypts her own input and computes an encryption of $g(x,y)$ using the homomorphic properties of the FHE scheme (i.e., using the eval algorithm). Alice then sends this encryption to Bob, who decrypts and now has the intended output.
Note that if you assume active adversaries this breaks down because you have no guarantee that Alice will compute the agreed upon function.
You would need to make sure that the ciphertext Alice sends to Bob does not reveal more information about Alices input than $g(x,y)$. As Ricky points out in the comment this may require some additional properties of the FHE scheme. One way to ensure this (as Ricky points out) is to require the FHE scheme to be circuit private. This means that an encryption of $g(x,y)$ obtained using the eval algorithm is indistinguishable from a fresh encryption of $g(x,y)$, i.e., an encryption obtained by directly encrypting the result $g(x,y)$.