The distinction is pretty straightforward, but it's best illustrated by example. I've written the same simple story in two different styles.
Genre fiction
A bullying dog next door annoys a cat. One day, the cat finds a magic amulet, and discovers that by rubbing its cheeks and drooling on it, it can transform into a dog! The cat decides to attack the dog and teach it a lesson. But when the cat creeps into the dog's yard, he finds that the dog is being bullied by an even bigger dog! Dog and cat-disguised-as-dog team up and defeat the bully dog, and make friends. Then the spell wears off and the cat returns to being a cat - and everyone learns a valuable lesson.
Literary fiction
A bullying dog next door annoys a cat. The cat spends time prowling back and forth, scratching himself with his own claws, worrying about the accounting business he runs. He falls asleep and dreams of chasing a ball of yarn, which whispers, "Wait!" in the voice of the mother he can barely remember. The cat wakes up and overturns his food bowl in rage. The dog arrives, blustering, and his loudness and shortness of breath remind the cat of a sergeant he knew when he was in the army. The grapple for a few seconds, then the passion ebbs and they slump against each other, overwhelmed with despair and ennui. In the next yard, a kitten and a puppy bark and tumble over each other. The cat, consumed with thoughts of mortality, yowls and rolls on the floor, hacking up the hairball that is his life. The dog slowly and mechanically digs his own grave. In a rotting wooden chair by the house, the cat's elderly owner reads the same letter from his dead daughter over and over again, and chuckles with mechanical emptiness. The sun slips below the horizon and a killing dark smothers their tortured souls with its vacuous respite.
Genre fiction
A bullying dog next door annoys a cat. One day, the cat finds a magic amulet, and discovers that by rubbing its cheeks and drooling on it, it can transform into a dog! The cat decides to attack the dog and teach it a lesson. But when the cat creeps into the dog's yard, he finds that the dog is being bullied by an even bigger dog! Dog and cat-disguised-as-dog team up and defeat the bully dog, and make friends. Then the spell wears off and the cat returns to being a cat - and everyone learns a valuable lesson.
Literary fiction
A bullying dog next door annoys a cat. The cat spends time prowling back and forth, scratching himself with his own claws, worrying about the accounting business he runs. He falls asleep and dreams of chasing a ball of yarn, which whispers, "Wait!" in the voice of the mother he can barely remember. The cat wakes up and overturns his food bowl in rage. The dog arrives, blustering, and his loudness and shortness of breath remind the cat of a sergeant he knew when he was in the army. The grapple for a few seconds, then the passion ebbs and they slump against each other, overwhelmed with despair and ennui. In the next yard, a kitten and a puppy bark and tumble over each other. The cat, consumed with thoughts of mortality, yowls and rolls on the floor, hacking up the hairball that is his life. The dog slowly and mechanically digs his own grave. In a rotting wooden chair by the house, the cat's elderly owner reads the same letter from his dead daughter over and over again, and chuckles with mechanical emptiness. The sun slips below the horizon and a killing dark smothers their tortured souls with its vacuous respite.