Inconsistent TOTK performance depending on day

THIS HAS BEEN SOLVED. SOLUTION WRITTEN DOWN BELOW.

There's some days when TOTK runs fine at 60 fps the whole time. Then there's other days when it runs at a weird-feeling 30-40 fps in the same areas, and in the same moments in the game. It's that performance drop you get when building shaders, but constant, and I check and it's not building anything and I'm not doing something new or I'm not in a new area. I'm not sure what could be causing it.

This is on PC, in the latest yuzu build. Using the settings TOTK Optimizer sets and locked at 60 fps.

Ryzen 5 7600x and RX 6700, 32 gb ram

Edit: When it was running at 60 fps the whole time, I decided to take screenshots of everything on the performance tab in the task manager. Today it's not running well so I took screenshots again and compared them to the ones from when it was running well and I found out yuzu's GPU usage is crazy on "bad" days. It goes down to 40%, then back up to 70-90% and does this constantly.

Edit 2: I'm starting to assume it's some AMD thing. Forcing maximum clocks makes the game run better on "bad" days.

Edit 3 and two solutions: It was an AMD thing. The GPU was not using enough clock speed for Yuzu specifically.

Solution 1:
Turning on the force maximum clocks option in Yuzu.

Solution 2:
Bumping up the minimum frequency in the Tuning tab in the AMD software. It's recommended to leave it at 100mhz less than the maximum frequency. This has absolutely no downsides in Yuzu or in any other software, since these values will only apply when using a 3D application.