The Known World in Slow-Mo

I am really struggling with this one...only 65 pages in, and I am dragging my feet. The premise is such a fascinating one, but the narrative keeps losing me, and every time I connect to one of the characters, the story shifts away from them. And by the time we come back, I've forgotten who they were. Anybody else have a problem with this one??