After more research and answers on forums the PTC resistor on the PCV/oil filter housing was introduced as a new suspect. Plan was to disconnect that once the problem starts happening and see if it makes a difference. At the same time I thought it might be worthwhile removing the intake manifold again and pre-heating the cam sensor, coils and anything else under there to see if I could induce the problem when the rest of the car was cold.
The pre-heating didn't do anything and there was no way I could reach the electrical connector on the oil filter housing without removing the airbox so I gave up on that.
Interestingly the car did something unusual this time, although I have seen this once before. On one restart it ran real rough and was idling too high but still up and down. While it was doing this the anti-skid light came on and an anti-skid error message came up. For a few starts it was still running rough like this and then on another restart it went back to its usual idle too low and stalling. I was recording a video at the time:
Since then I removed the CEM to check for bad solder joints. So far I've only got it out and removed the back. Next step is to remove all of the fuses so I can get the circuit board out.