[subject] + 难以 + [verb]
让人 + 难以 + [verb]
难以 before a verb says something is hard to do – often so hard it barely happens. The verb is almost always two syllables: 相信, 理解, 接受, 忘记, 解决. It belongs to writing and careful speech.
让人难以 + verb, "makes it hard to…", is the frame you'll meet most.
很难 + verb is the everyday form and takes any verb. 难以 is the formal one and wants a two-syllable verb.
A few everyday words fuse 难 with a one-syllable verb – 难吃, 难看, 难听. Those are adjectives with their own point.
难以 already carries "hard"; a degree word in front of it is one too many.
With a one-syllable verb it sounds broken. Switch to 很难 or to the fused adjective.