Czech is just as difficult as Polish (if not more so for English speakers - ř is a dreadful letter and far more difficult than rz for English speakers)
Czech is also difficult for Polish speakers exactly because it bears so many similarities to it. When I have recently started to get some linguistic insight into Czech (mluvitecesky.net - multilingual website to study Czech), I have soon discovered that I'm forgetting the proper Czech pronounciation of words as soon as I leave the website. This is because my learned Polish pronounciation of a very similar, almost identical, Czech word immediately takes over the Czech one.
ř is a dreadful letter and far more difficult than rz
This may be so for an English speaker. When I listen to ř as an isolated sound, I can hear a light "r" followed by ž which ž is perhaps pronounced a bit lighter than the Polish "ż". But when they start to pronounce ř in words and phrases talking at a normal speed, this light "r" tends to disappear and what I can hear is pure ž.
Notice that in Polish the sound of "rz" was also different to the sound of "ż" several centuries ago. That's why we have the two different spellings of what is now the identical sound "ż".
Notice also that the Polish spellings cz, rz, sz were originally Czech invention which the Polish imported, but which Jan Hus, reformer of the Czech ortography got rid of in Czech, replacing them with č, ř, š. Those mediaeval Czech spellings, however, have survived in Polish.