There is formal language and then there is informal, casual language.
For the school classrooms you need formal, i.e., ‘academic’ language. For the lunch room and with your friends you need informal/casual language skills.
At work, especially if it involves an office, boardroom and the like, you need formal i.e., ‘business’ language skills. Each job and career comes with a plethora of ISV (Industry-Specific Vocabulary), acronyms and lingo that you must learn to be successful in your career advancement and to obviously understand the fast-paced conversations going on around you.
Most people who speak ESL (English as a Second Language) focus on formal language skills first, then ‘graduate’ to casual conversation, then if required learn Business English.
Do what you have to do to fit in, understand those around you, and succeed in your industry.
You might have a high TOEFL or TOEIC score, but how’s your understanding of casual small talk and chit-chat? Do you follow English-speaking TV and movies easily? Do you follow casual conversations with your friends and co-workers well or do you get left out when they are not speaking about technical stuff?
It’s probably time to evaluate your informal English language skills and maybe study more pop culture and more idioms.
Here are some random sites to learn idioms (there are many more out there!):
There are textbooks, Podcasts and eBooks too.
I put together an eBook of idioms! Here are the links, in case you want to check it out:
Everyday English Idioms
Barnes & Noble NOOK: