I think the difficulty of the course comes from the sheer amount of content and the questions that come in the exams. I don't find any of the concepts particularly difficult or confusing, however, I really struggle with difficult exam style questions where their application comes into place.
Some classes are notoriously hard such as Mathematics HL and Physics HL, so making sure to pick the right HL subjects is important for your overall success in IB. Just as importantly, SL classes need to be "easy" points and shouldn't create as much of a burden as HL classes.
VZ3pC3U.