Builds ran from Jenkins pipeline fail while builds ran manually on agent machine succeed

Using Jenkins Ask a question
1

I am running into issues when I try to run a python build script from a Jenkins Pipeline, which I don’t get when I run the same script on the same machine outside of Jenkins. I am wondering if my approach has some obvious flaws or if someone has some insights into the issue I am running into.

My Jenkins setup looks like this:

  • I have a windows jenkins controller
  • The controller uses the EC2 plugin to allocate and spin up build agents also running on windows
  • The controller connects to the agent using Winrm
  • The Build agents are running on a custom windows AMI i have put together that contains everything needed to run UnrealEngine builds and I log in using a user that has all of the required permissions to run the builds
  • When a build is triggered, the controller spins up an agent on a ec2 machine, which pulls down the project, runs the build and then uploads the completed builds to an S3 bucket

I have a Python build script that runs the various steps of my UE4 builds that I can start through powershell. If I run this script on the EC2 machine manually, the builds work just fine without issues. When I use the same powershell command in my Jenkins pipeline script to build the project, I run into issues that I never get outside of jenkins.

The error looks like this:

c1xx: error C3859: Failed to create virtual memory for PCH  
c1xx: note: the system returned code 1455: The paging file is too small for this operation to complete.           
c1xx: note: please visit [https://aka.ms/pch-help](https://aka.ms/pch-help) for more details  
c1xx: fatal error C1076: compiler limit: internal heap limit reached

The errors seem to change depending on the run, but always seem to concern memory limits, which the instance shouldn’t have. I spin up instances with 64gb of physical memory and I’ve assigned a 64gb of virtual memory (paging file size) as well. Again, the thing that confuses me is the lack of this issue when running the build manually on the agent machine.

I’ve looked into this for the last couple of days and I haven’t gotten closer to a solution. Based on what I’ve found, you can adjust the JVM memory limit using the -Xmx and -Xms options.

I’ve set those to these values: -Xmx16G -Xms8G which should be plenty to run the builds but i still run into the same issue.

I feel like I am missing something obvious. I am relatively new to using Jenkins so any insights into what could be happening here would be greatly appreciated! I would love to learn more about the differences of running a build on the machine vs what happens with a jenkins build. I am more than happy to provide any additional info that could be useful.

2

Some additional info:

Java version on the agent:

java 17.0.6 2023-01-17 LTS
Java(TM) SE Runtime Environment (build 17.0.6+9-LTS-190)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.6+9-LTS-190, mixed mode, sharing)

The memory utilisation on the machine during the build step that fails is barely above 10%. Both during the jenkins step and when called manually

3

if you are running things through shell/bat

bat(‘python -v’) it shouldn’t matter how much memory java has, its a seperate process

I guess what are you doing?

4

thanks for getting back to me. I have managed to get past this issue today.
The issue stemmed from the winrm connection and its MaxMemoryPerShellMB configuration.

The Ec2 plugin documentation suggests setting this value to a specific mb value, but setting it to 0 as this stack overflow post suggests has sorted the issue: c++ - C3859: Virtual memory range for PCH exceeded - Stack Overflow

5

I’m sorry I don’t know much about jenkins+windows

I’ve heard from @slide_o_mix that ssh on windows is the way to go these days? I think

6

I think the EC2 plugin is what uses WinRM. I am not sure if it allows you to use ssh, but that is deft the way to go if possible.

7

From looking online, if you configure the node as a unix node and just use forward slashes in paths, e.g., c:/blah/blah, it should work with ssh.

8

great to know, I’ll have to try that in the near future. Thanks for all the info!