The HandleRFBServerMessage function in libvncclient/rfbproto.c in LibVNCServer 0.9.9 and earlier does not check certain malloc return values, which allows remote VNC servers to cause a denial of service (application crash) or possibly execute arbitrary code by specifying a large screen size in a (1) FramebufferUpdate, (2) ResizeFrameBuffer, or (3) PalmVNCReSizeFrameBuffer message.
lists.opensuse.org/opensuse-updates/2015-12/msg00022.html
seclists.org/oss-sec/2014/q3/639
secunia.com/advisories/61506
secunia.com/advisories/61682
ubuntu.com/usn/usn-2365-1
www.debian.org/security/2014/dsa-3081
www.ocert.org/advisories/ocert-2014-007.html
www.openwall.com/lists/oss-security/2014/09/25/11
www.oracle.com/technetwork/topics/security/bulletinoct2015-2511968.html
www.securityfocus.com/bid/70091
github.com/newsoft/libvncserver/commit/85a778c0e45e87e35ee7199f1f25020648e8b812
lists.debian.org/debian-lts-announce/2019/10/msg00042.html
security.gentoo.org/glsa/201507-07
usn.ubuntu.com/4587-1/