Looking for some help here with my I2C bus - I have an Olimex ESP32-PoE-ISO and two DFR0626 GPIO expander modules (though I've parred back to one for now). On the one connected expander I currently have 8 pins configured as outputs and 8 pins configured as inputs. Input detections seem to work fine, but when I send triggers to the output pins I get about a 50% success rate with either the desired output or a couple of different error messages:

[16:24:18][W][i2c:073]: Received NACK on transmit of data for address 0x20
[16:24:18][W][i2c:086]: Requesting 1 bytes from 0x20 failed!

[16:24:23][W][i2c:073]: Received NACK on transmit of data for address 0x20
[16:24:23][W][i2c:076]: Unknown transmit error 5 for address 0x20

My wiring is from the ESP's UEXT connector using pins 3.3v, GND and GPIO13 to SDA and GPIO16 to SCL.
If I modify my ESPHome config to only configure the input pins or only configure the outputs pins then they work as expected. Problems arise when I have inputs and outputs configured at the same time. Any ideas what the problem here may be?