Console log parser no longer working on some jobs "java.nio.charset.MalformedInputException: Input length = 1”

Using Jenkins Ask a question
1

We’re using

  • Jenkins 2.375.2 / 2.361.4
  • jdk-11.0.12+7 / jdk-11.0.17+8
  • Log Parser Plugin Version 2.3
  • All machines are running Linux (RH7)

We had been using Jenkins 2.361.4 and JDK 11.0.12+7. With that combination the console log parser just worked as expected.
We then updated first Jenkins to 2.375.1, later to 2.375.2. Sometimes after that we updated the used JDK from jdk-11.0.12+7 to jdk-11.0.17+8.

After that we saw by chance a few failed console parser runs, the parsed console just contained:
“ERROR: Failed to parse console log
java.nio.charset.MalformedInputException: Input length = 1”
Other jobs/runs had the correct parsed console. We attributed this to some hiccup and continued with the pressing daily tasks.
Then our users started to complain about missing parsed consoles (we rely on this to speed up analysis)

I found the error message in these cases in build.xml of the affected runs:

<hudson.plugins.logparser.LogParserAction plugin=log-parser@2.3.0>
  <build class="build" reference="../../.."/>
  <result>
    <totalErrors>0</totalErrors>
    <totalWarnings>0</totalWarnings>
    <totalInfos>0</totalInfos>
    <totalDebugs>0</totalDebugs>
    <totalCountsByExtraTag/>
    <linkedFilesByExtraTag/>
    <extraTags/>
    <failedToParseError>java.nio.charset.MalformedInputException: Input length = 1</failedToParseError>
  </result>
</hudson.plugins.logparser.LogParserAction>

Grepping for this, I found a lot of cases – but by far not all jobs.
On a trial-system (a rsynced copy of the current state), I downdated Jenkins (2.375.2 → 2.361.4) and JDK (jdk-11.0.17+8 → jdk-11.0.12+7).
I expected this to solve the problem, because these had been the “obvious” changes. I hoped I can find some explanation in the release note of the causing component.

But to my amazement, the problem was still there. (We log ‘java –version’ to the startup log; the Jenkins version is visible anyway); both had been like expected)

So, I went hunting for the source of the problem. Searching for the error message pointed me to encoding problems for the chars.
The beginning of one example of failing console log looked like this:

“Started by user ^[[8mha:////4I7 …”
It seems these “ha:////” (resembling a URL) are used to “encode” links to users and triggering jobs.
With the “^[“ being a non-ASCII char - but then we found the same char over and over, also on jobs with working parsed console.

The first time the problem appeared, correlates to the (minor) java update listed above. But as said, reverting this, didn’t solved the problem.

Questions:

  • Ideas how we can get our parsed console back ?
  • Why did undoing jdk and Jenkins core, didn’t revert the problem ? What else do I need to consider ?
  • Any ideas, how for further debug ?

Thanks

Martin

1 Like
2

Hello @martinjost,

Did you install the updates on a new set of servers, or are you using the same servers you used to work on with previous versions of Jenkins?
If you’re working on new servers, how does the installation differ from the previous ones?
If you’re still on the same servers, what else did change? I guess you update your OS from time to time, so is there anything in the changelog related to charsets or encodings?

How did you upgrade Jenkins? Via sudo yum upgrade ?
How did you downgrade Jenkins?
Thanks.

3

Hello Bruno,
thanks for looking into this and asking back !

We have two machines, which are suitable to act as controllers. We rsync the state of the active machine to the standby one, once per day (by a CI job (*)). I used that standby system to do my tests.
(Obviously I need to take out the build nodes, which are used by the active one and similar adaptations)

Besides that we have a"real" backup and we commit the state of Jenkins (job.xml files, Jenkins config files and surrounding scripts) automatically to GIT once a day. (What we didn’t commit manually after changes on the command line) These commits include the java version used for jenkins and the jenkins version itself, in log-files committed as well.

“Did you install the updates on a new set of servers, or are you using the same servers you used to work on with previous versions of Jenkins?”

See above - a “new” server. One agent moved from production system to the test system.

“If you’re working on new servers, how does the installation differ from the previous ones?”

The installation itself, shouldn’t differ at all. (Same HW, same RH 7 installed; should be patched to the same state)
Obviously we could have slips here. But in the past we had been changing several times the active/standby role between the two machines, without seeing issues popping up.

If you’re still on the same servers, what else did change? I guess you update your OS from time to time, so is there anything in the changelog related to charsets or encodings?

On the active system, nothing “obvious” (manually, intentional) changed between “working” and “not working” besides JDK and jenkins.war used.
We checked the changes visible in GIT (see above) for the Jenkins config, without finding something suspicious. (I had expected that maybe some plugin/jenkins/JDK update, caused Jenkins to write a modification to some config file(s) [we had seen this in the past], so in these cases “just” undoing the manually done change, wouldn’t be enough to “really” revert it. But nothing like that showed up.
For OS updates our central IT is responsible. But at least for changes that require a reboot, they need (and ask) for our confirmation. (Besides “Global maintenance break” weekends, which get announced) No such update (we’re aware of) happened between “working” and “not working”.

“How did you upgrade Jenkins? Via sudo yum upgrade ?”

No, We don’t have general root rights on the machines. (The base system is managed by our IT)
So we have Jenkins installed on local FS on each machine, using the jenkins.war files. Similar for the used JDK.
We keep the currently used and the previously successfully used jenkins.war and JDK, both filenames include the version numbers.
We have generic ‘jenkins.war’ and ‘jdk’ symlinks, which “point” to the jenkins.war and JDK to be used by all our scripts referring to these (including the Jenkins startup script)
If we want to update Jenkins itself or JDK, we install the new version, with the version number included and make the generic symlinks point to these.
On updating we keep the last version. So it is easy to switch back, by simply changing the symlinks again. The JDK version used is logged (from a ‘java --version’) call to the startup log; similar for the Jenkins version used - these go to the GIT commit each day. So we know, what versions are actually in use or - via GIT - have been used before.
(This mechanism we use for years now, without problems; no “big” changes done here for a long time)

“How did you downgrade Jenkins?”

We had used that symlink-mechnism.for the update and I used it to “downdate” on the standby system.

Thanks.for looking into this - feel free to ask additional questions

Martin

(*) yes, so while we do this rsync, the active system is running and the state might change.
To limit possible inconsistencies, we run two rsyncs back to back, so the second one only needs to rsync, whatever changed during the first run. Also we trigger that job during the night (less activities) and while no other time scheduled job is running.
If we do an “orderly” swap of the machines, we manually run this rsync, with no Jenkins running, to be sure to have the same state.

4

Your post helped me get pointed in the right direction to solving a very similar issue after upgrading to 2.375.3 and subsequently JAVA 8 → 11. Seems that encoding is handled a bit differently between JAVA versions. We had a GlobalAssemblyInfo.cs file which was errantly encoded in ANSI where all others were UTF-8. This had been building fine. However after the upgrade, we began seeing the same error as you reported inside of the Change Assembly Version step of our process via the change-assembly-version-plugin plugin. Inside of the .cs file was a copyright character that was now causing the issue. Once we changed encoding on the .cs file we were good. Hope this helps!