The HandleRFBServerMessage function in libvncclient/rfbproto.c in LibVNCServer 0.9.9 and earlier does not check certain malloc return values, which allows remote VNC servers to cause a denial of service (application crash) or possibly execute arbitrary code by specifying a large screen size in a (1) FramebufferUpdate, (2) ResizeFrameBuffer, or (3) PalmVNCReSizeFrameBuffer message.
https://usn.ubuntu.com/4587-1/
https://security.gentoo.org/glsa/201507-07
https://lists.debian.org/debian-lts-announce/2019/10/msg00042.html
http://www.securityfocus.com/bid/70091
http://www.oracle.com/technetwork/topics/security/bulletinoct2015-2511968.html
http://www.openwall.com/lists/oss-security/2014/09/25/11
http://www.ocert.org/advisories/ocert-2014-007.html
http://www.debian.org/security/2014/dsa-3081
http://ubuntu.com/usn/usn-2365-1
http://secunia.com/advisories/61682
http://secunia.com/advisories/61506
http://seclists.org/oss-sec/2014/q3/639
http://lists.opensuse.org/opensuse-updates/2015-12/msg00022.html