Fix integer overflow computing glyph_size in red_get_string

If bpp is int the formula can lead to weird overflows. width and height
are uint16_t so the formula is:

  size_t = u16 * (u16 * int + const_int) / const_int;

so it became

  size_t = (int) u16 * ((int) u16 * int + const_int) / const_int;

However the (int) u16 * (int) u16 can then became negative to overflow.
Under 64 bit architectures size_t is 64 and int usually 32 so converting
this negative 32 bit number to a unsigned 64 bit lead to a very big
number as the signed is extended and then converted to unsigned.
Using unsigned arithmetic prevent extending the sign.

Signed-off-by: Frediano Ziglio <fziglio@redhat.com>
Acked-by: Christophe Fergeau <cfergeau@redhat.com>

0 files changed, 0 insertions, 0 deletions