Minigames are essentially small games within games. Some consist of semi-normal gameplay to accomplish some non-normal task, others involve changing the gameplay entirely. Sometimes they are optional, sometimes they are not as optional. Minigames, when exectuted well, can be fun, but are sometimes just annoying.
The key to keeping a minigame from being annoying is to really give the player a choice in the matter and to have a reward in the end. I've played many an annoying minigame, but they were optional. For instance, there were several minigames in The Legend of Zelda: Twilight Princess, one such minigame had the player tilting the Wii Remote to carefully guide a marble into a hole. Not a complicated minigame, but slightly annoying, and still I spent a good amount of time getting past every level. Why would I play an optional minigame when I could be playing Twilight Princess proper? Because there was a reward, and it didn't seem like I had to. Same thing goes for some more mandatory minigames, but the reward is usually advancing the plot or enhancing a character's stats. The reward gives players a reason to spend hours mastering the minigame's mechanics.
Bad minigame implementation however can lead to frustration. Mandatory minigames can be a problem because they may be poorly designed, but the minigame must be completed to advance the game, thus forcing the player to reach a standstill in the game. At this point, the frustration is exacerbated, maddening the player. Usually though, developers are smart enough that only the first few levels of a multi-level mingame must be completed, with additional levels becoming available later in the game. Also, games that use minigames to improve a characters stats, like Final Fantasy XII's license board system, can be frustrating just because you have to use it. The license board required License Points, or LP, to use, a small amount of which could be gained by defeating monsters. This causes many people to spend hours slowly gaining LP, which can be very boring. A gamer does not want to be bored playing a game.
Minigames can be incredibly frustrating, but, if implemented well, can be less of a nusence, and more of a fun diversion.