Jenkins compromised

Hello,

I inherited a Jenkins 2.319.2 installation and noticed a continuously running job which downloads and launches crypto-miners. I presume it’s been created remotely.

Is there a way to tell from the logs or other indicators if this is
a) the result of a security exploit in Jenkins or
b) compromised credentials?

If so, how?

The instance is open to the internet but obviously requires login/password, some API tokens have been configured to invoke jobs remotely.

What would the general advise be here? Update Jenkins and all plugins to their latest version?

Thank you,

Best regards,

David

Logs from the time the job was created/launched below:

Jun 14, 2022 1:09:19 AM WARNING org.eclipse.jetty.server.session.SessionHandler getSession
Invalidating session node0ygxkzxqixxkz1lhn04ww88avf5669 found to be expired when requested
Jun 14, 2022 1:09:19 AM WARNING org.eclipse.jetty.server.session.SessionHandler getSession
java.lang.IllegalStateException
	at org.eclipse.jetty.server.session.Session.beginInvalidate(Session.java:993)
	at org.eclipse.jetty.server.session.Session.invalidate(Session.java:934)
	at org.eclipse.jetty.server.session.SessionHandler.getSession(SessionHandler.java:918)
	at org.eclipse.jetty.server.session.SessionHandler.getHttpSession(SessionHandler.java:553)
	at org.eclipse.jetty.server.session.SessionHandler.checkRequestedSessionId(SessionHandler.java:1673)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)

Jun 14, 2022 1:09:19 AM WARNING hudson.security.csrf.CrumbFilter doFilter
Found invalid crumb 66732d316dd99ee18809c9c787c037c567badcecfe46c10e681513aec3faf916. If you are calling this URL with a script, please use the API Token instead. More information: https://www.jenkins.io/redirect/crumb-cannot-be-used-for-script
Jun 14, 2022 1:09:19 AM WARNING hudson.security.csrf.CrumbFilter doFilter
Found invalid crumb 66732d316dd99ee18809c9c787c037c567badcecfe46c10e681513aec3faf916. If you are calling this URL with a script, please use the API Token instead. More information: https://www.jenkins.io/redirect/crumb-cannot-be-used-for-script
Jun 14, 2022 1:09:19 AM WARNING hudson.security.csrf.CrumbFilter doFilter
No valid crumb was included in request for /ajaxExecutors by worker. Returning 403.
Jun 14, 2022 1:09:19 AM WARNING hudson.security.csrf.CrumbFilter doFilter
No valid crumb was included in request for /ajaxBuildQueue by worker. Returning 403.
Jun 14, 2022 1:09:45 AM WARNING hudson.model.AbstractProject submit
label assignment is using legacy '_.assignedLabelString'
Jun 14, 2022 1:20:12 AM INFO hudson.model.AsyncPeriodicWork lambda$doRun$1
Started Periodic background build discarder
Jun 14, 2022 1:20:12 AM INFO hudson.model.AsyncPeriodicWork lambda$doRun$1
Finished Periodic background build discarder. 4 ms
Jun 14, 2022 2:20:12 AM INFO hudson.model.AsyncPeriodicWork lambda$doRun$1
Started Periodic background build discarder
Jun 14, 2022 2:20:12 AM INFO hudson.model.AsyncPeriodicWork lambda$doRun$1
Finished Periodic background build discarder. 4 ms
Jun 14, 2022 3:20:12 AM INFO hudson.model.AsyncPeriodicWork lambda$doRun$1
Started Periodic background build discarder
Jun 14, 2022 3:20:12 AM INFO hudson.model.AsyncPeriodicWork lambda$doRun$1
Finished Periodic background build discarder. 3 ms
Jun 14, 2022 4:20:12 AM INFO hudson.model.AsyncPeriodicWork lambda$doRun$1
Started Periodic background build discarder
Jun 14, 2022 4:20:12 AM INFO hudson.model.AsyncPeriodicWork lambda$doRun$1
Finished Periodic background build discarder. 3 ms
Jun 14, 2022 4:31:57 AM INFO hudson.model.AsyncPeriodicWork lambda$doRun$1
Started Workspace clean-up
Jun 14, 2022 4:31:57 AM INFO hudson.model.AsyncPeriodicWork lambda$doRun$1
Finished Workspace clean-up. 1 ms
Jun 14, 2022 5:20:12 AM INFO hudson.model.AsyncPeriodicWork lambda$doRun$1

Destroy the machine and start again. Once a machine is compremised you have to assume there’s lots of hidden junk.

Review Security Advisories maybe? I would recommend reaching out to a vendor as open source volunteers probably can’t give you the attention you need to dig into this deeply. Also assume all credentials are compromised (see my first point)

1 Like