I'm starting to think that's the issue. Looks like factory oil pressure specs are as low as 4psi at idle, and the issue is only at idle. I was idling a little low but I turned it up to 750, checked timing, and checked everything ignition related.
I'm going to change the oil to a 10w40 conventional and I may pick up the oil pressure regulator from LC engineering to see if that smooths everything out. It's a pretty cheap part for them, 27 bucks.