Have heard that some use a concoction involving mineral oil. Which doesn’t have much grit in it.
Use only a wet, "grabby" cloth, such as a heavy tissue paper or microfiber washcloth, to clean your instrument. A luthier may elect to use shellac or an abrasive paste to bring a shine back to your instrument, but no layman has business applying those methods.
Walnut oil is a “drying” oil, which means it polymerizes as it oxidizes. Consequently, it can leave a sticky residue if it is not wiped away thoroughly.
I’ve defer to David Burgess and the other luthiers on violin varnish compatibility.
My guess is you’re leaving very little residue here.
One of these days, I’ll do some research on the Hansen solubility parameters of various violin varnishes. There’s too much black art and too little science. I like Bob Flexner’s perspective on this stuff.
Violinist.com is made possible by...
International Violin Competition of Indianapolis
Violinist.com Holiday Gift Guide
Dimitri Musafia, Master Maker of Violin and Viola Cases
Johnson String Instrument/Carriage House Violins
Discover the best of Violinist.com in these collections of editor Laurie Niles' exclusive interviews.

Violinist.com Interviews Volume 1, with introduction by Hilary Hahn

Violinist.com Interviews Volume 2, with introduction by Rachel Barton Pine
nut, dust/bits ground into the varnish of your violin.