What is your biggest challenge learning English? Vocabulary? Grammar? Listening? No. I don't think so. I think your biggest challenge is that you want to understand English perfectly.
If I say something too fast, or if I use words and phrases that my students don't know, they can get annoyed or frustrated. If a student tries to speak and can't communicate their idea perfectly, they can get annoyed or frustrated. In my lessons, my students are often not happy until they have a perfect understanding of every word and sentence in our textbook. If they cannot understand it all, some can get annoyed and frustrated.
Why? What's the hurry? What's so bad about only kind of getting it? Why do you want to be perfect? When you think about it, wanting to understand language perfectly is kind of crazy.
First of all, when you read or listen to your first language, do you understand it perfectly? I think that sometimes you don't. Sometimes, you don't catch what someone says on TV because they speak too fast, or they mumble, or it just doesn't make sense. Sometimes, someone will use a word you don't know and you have to ask them what that new word means. But in most of these situations, you would kind of understand (なんとなく分かる). You would say to yourself, "I think she said..." or "I don't know exactly what that word means, but I guess it's..."
Secondly, and much more importantly, native speakers do not understand English the way you want to understand English. Let's test this idea. Imagine that I am your student, and we are studying Japanese. I ask you a question:
"In the sentence, "やればやるほど楽しくなる", what does 「ほど」 mean?"
Can you answer that question? I doubt it. 「ほど」 is a pretty strange thing.「ほど」is「ほど」 In fact, all of that sentence is strange for an English speaker. We would say, "The more you do it, the more fun it is" in English And I bet that English pattern is strange for you. What does "the more... the more" mean? Doesn't it make more sense to say, "You do it more, it is more fun"?! Probably, but native speakers don't say it like that. Why? I honestly have no idea. Zero. None. Probably something about switching things around and emphasis. But who really cares? It is what it is.
Take a simpler example. When I first learned Japanese, I learned how to say who I was:
私はマシューです。
Simple, right? I wanted to understand this sentence perfectly. I asked my teacher, Mr Ito:
"Sir, what is 'watashi'?"
"That means 'I'."
"I see. And I guess that Mashuu is how to say my name. Does wa mean "am"? If it is Watashi wa Mashuu desu, then it means 'I am Matthew'..."
"Well, no. Desu means 'am', kind of."
"So if desu means 'am', then what is wa?!"
So what is は? You know how to use は, but I don't think you could tell me what it means in any perfect and easy-to-understand way. But you don't need to be able to say what something means to use it well. (That is what a dictionary is for, and it is also why people who can't use dictionaries correctly speak bad English, but that is another story.) It's the same with lots of words. What does "for" mean? For native speakers, "for" is "for". And for you, は is は.
It is true that sometimes you can think about things and understand them deeply. You should always try. You can find rules and patterns that will help you make sentences and communicate. But sometimes, you have to relax. The danger is that you start thinking too deeply about stuff. It makes you worry when you do not understand. You don't need to understand everything now. You can take your time. Leave the difficult things for later. Maybe you don't have enough experience to understand something now, but in a month or a year you will think, "I got it!" Right now you have to just remember it. You have to be happy with being able to kind of get it, because often that's all you need!