Graph section changing color depending on view. How to fix?

For my charts when I change views the different section change color, it can be really confusing.

Here's an example:

Those images show one graph in three different view. As you can see some times the yes value can be blue and some times it's orange. How do I fix that so one value is always the same color.

Thanks