Mitigating error IpynbDiff::InvalidTokenError

Sentry: https://sentry.gitlab.net/gitlab/gitlabcom/issues/3204685/

This error happens on the jsonparser that tokenizes the Notebook
file, but it happens very rarely and it's hard to reproduce. As a
mitigation, when users encounter this problem, we will fallback to
classic raw diffs.

lib/gitlab/diff/custom_diff.rb

@@ -10,14 +10,12 @@ module Gitlab
             transformed_for_diff(new_blob, old_blob)
             Gitlab::AppLogger.info({ message: 'IPYNB_DIFF_GENERATED' })
           end
-        rescue IpynbDiff::InvalidNotebookError => e
+        rescue IpynbDiff::InvalidNotebookError, IpynbDiff::InvalidTokenError => e
           Gitlab::ErrorTracking.log_exception(e)
           nil
         end
 
         def transformed_diff(before, after)
-          Gitlab::AppLogger.info({ message: 'IPYNB_DIFF_GENERATED' })
-
           transformed_diff = IpynbDiff.diff(before, after,
                          raise_if_invalid_nb: true,
                          diffy_opts: { include_diff_info: true }).to_s(:text)

lib/gitlab/diff/rendered/notebook/diff_file.rb

@@ -57,7 +57,7 @@ module Gitlab
               IpynbDiff.diff(source_diff.old_blob&.data, source_diff.new_blob&.data,
                              raise_if_invalid_nb: true,
                              diffy_opts: { include_diff_info: true })
-            rescue IpynbDiff::InvalidNotebookError => e
+            rescue IpynbDiff::InvalidNotebookError, IpynbDiff::InvalidTokenError => e
               Gitlab::ErrorTracking.log_exception(e)
               nil
             end