Commit 0bd14972 authored by Robert Speicher's avatar Robert Speicher

Merge branch 'fix/encoding-helper-clean-blowing-up-on-utf-16be' into 'master'

Fix EncodingHelper#clean blowing up on UTF-16BE strings

Closes gitaly#1101

See merge request gitlab-org/gitlab-ce!17943
parents e583524d 1f2cc29c
...@@ -90,7 +90,7 @@ module Gitlab ...@@ -90,7 +90,7 @@ module Gitlab
end end
def clean(message) def clean(message)
message.encode("UTF-16BE", undef: :replace, invalid: :replace, replace: "") message.encode("UTF-16BE", undef: :replace, invalid: :replace, replace: "".encode("UTF-16BE"))
.encode("UTF-8") .encode("UTF-8")
.gsub("\0".encode("UTF-8"), "") .gsub("\0".encode("UTF-8"), "")
end end
......
...@@ -161,6 +161,11 @@ describe Gitlab::EncodingHelper do ...@@ -161,6 +161,11 @@ describe Gitlab::EncodingHelper do
'removes invalid bytes from ASCII-8bit encoded multibyte string.', 'removes invalid bytes from ASCII-8bit encoded multibyte string.',
"Lorem ipsum\xC3\n dolor sit amet, xy\xC3\xA0y\xC3\xB9abcd\xC3\xB9efg".force_encoding('ASCII-8BIT'), "Lorem ipsum\xC3\n dolor sit amet, xy\xC3\xA0y\xC3\xB9abcd\xC3\xB9efg".force_encoding('ASCII-8BIT'),
"Lorem ipsum\n dolor sit amet, xyàyùabcdùefg" "Lorem ipsum\n dolor sit amet, xyàyùabcdùefg"
],
[
'handles UTF-16BE encoded strings',
"\xFE\xFF\x00\x41".force_encoding('ASCII-8BIT'), # An "A" prepended with UTF-16 BOM
"\xEF\xBB\xBFA" # An "A" prepended with UTF-8 BOM
] ]
].each do |description, test_string, xpect| ].each do |description, test_string, xpect|
it description do it description do
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment