Lockup When Changing Screenmodes

Recently my Radeon 9250 started having weird glitches and corruption, so I swapped it out for another Radeon 9250. Glitches are gone and it seems to work fine, however when I change screenmodes in Screenmode preferences, the system locks up, with an almost completely blank grey screen but for the frozen mouse pointer and a few corrupted lines from the Workbench screen. I don't think this is a hardware issue because testing the screenmode works just fine, and other software opening different screenmodes also works just fine. This only happens when saving or using the screenmode for Workbench. If I save the mode and reboot, the saved mode works just fine too.

Any ideas? Is there possibly some confusion between the old display modes and the new perhaps, given that both cards have the same name but are physically different models? Is there a way to flush the screenmodes database? Renaming screenmodes.prefs resulted in a black screen after the splashscreen, so I needed to boot from the 4.1FE CD to restore the file.

There is no such thing as a database for screenmodes.
All modes available will be stored as tooltypes in your monitor prefs file.

screenmode.prefs just hold the information which of the monitor modes is used on start.

That sounds more like a software in the background interfering to me.

Since it works after a reboot, but not while you are on Workbench i'd go from cutting down running apps (see WBStartup and the startup-sequences) and cornering the bad boy.

Also, if you boot with your OS4 CD and "USE"ing a different screenmode does NOT crash it's no hardware issue.

Thanks for the reply. There is a database of available screenmodes kept by Amiga OS, I'm just not sure whether it's generated on the fly at boot or whether it's persistent somewhere and needs to be wiped.

I've already ruled out any interference from software at startup, and yeah, I'm pretty sure it's a software thing - it doesn't happen with the old card.

Interestingly, it worked for a short while when I restored the settings to automatically detect settings and used a different monitor to force it to show different choices, then switched back to my normal monitor. But the automatic detection doesn't show all the modes I want to use, and switching back to the tooltype method causes the freezes again. At that point, switching back to automatic detection does not resolve the issue, so it could just have been a coincidence.

@Daedalus
It's generally the monitor that have the database DDC EDID and communicates with the card which resolutions and refresh rates it supports. Have you checked that your monitor is healthy? check with a ps4 or something similar with a hdmi connection (a DP connection is always better,but... ), delete the monitor.info and create a new one with screenmode (especially if you have created your own screen modes)? someone with more experience might know more about this I have only done it once when I upgraded to the RX580, and I didn't really see any difference on which modes there was.

The monitor is just fine, and the EDID information is correctly read. The custom modes I'm looking for aren't anything special, e.g. 1920x1080 & 1280x800, and they're selected from the options available in the Monitors tab of Screenmodes. It looks like an issue with the driver trying to reinitialise the graphics card and failing, as it doesn't even get to the point of attempting to use the new resolution; no resolution switch is detected by the display.

A difference that may be relevant between this card and my previous one is that the old one was a 256MB card, whereas the new one is 128MB. I can't see that being an issue however - OS4 correctly detects the amount of available RAM, and the issue also happens when switching to smaller screenmodes.