It may be a bit biased, but you should watch it unless you're someone who is
obsessed with the mainstream pirates, zombies, ninjas, robots, fighting, action, blood, horror, etc. etc. If you're someone who wants more of a drama that makes you feel and seeing good things happen and such (and not-so-good sad moments) then watch it. Anyone who doesn't watch it... Their loss.
If you want to see what everyone is talking about but don't want to sit through it, I
think you can just start with After Story, although you won't understand the characters as much. Mainly Nagisa and Fuko, but I think it's still watchable from After Story to the end.
EDIT: If you're interested, these are the sort of things that you'll see later on in ~After Story~.