Because Luffy-sama ate all the candy, and told Big Mam-sama that he would protect the island from now on, and Jimbei-sama is part of Luffy-sama's crew now and he told them to go on ahead, he'd catch up later.

[A storytelling prodigy, that's Shirahoshi.]