Could not start a new session. Response code 500. Message: unknown error: Chrome failed to start: crashed for Python app in Jenkins

I am currently facing challenges with running Selenium automation tests in a Jenkins environment.
I have a scheduled jenkins job which run in every five minute and executes the UI automated test cases written in Selenium.
I have written selenium test cases in Python with following versions:-
Selenium: 4.26.1 Selenium 4.26.1

Error message is as shown below:-

selenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome failed to start: exited normally.
E         (session not created: DevToolsActivePort file doesn't exist)
E         (The process started from chrome location /root/.cache/selenium/chrome/linux64/130.0.6723.116/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
E       Stacktrace:
E       #0 0x55af503386da <unknown>
E       #1 0x55af4fe4cf80 <unknown>
E       #2 0x55af4fe82db1 <unknown>
E       #3 0x55af4fe7e6c5 <unknown>
E       #4 0x55af4fec9cf9 <unknown>
E       #5 0x55af4fec9346 <unknown>
E       #6 0x55af4febd953 <unknown>
E       #7 0x55af4fe8c72e <unknown>
E       #8 0x55af4fe8d79e <unknown>
E       #9 0x55af50303f1b <unknown>
E       #10 0x55af50307eb8 <unknown>
E       #11 0x55af502f142c <unknown>
E       #12 0x55af50308a37 <unknown>
E       #13 0x55af502d5fef <unknown>
E       #14 0x55af50326ad8 <unknown>
E       #15 0x55af50326ca0 <unknown>
E       #16 0x55af50337556 <unknown>
E       #17 0x7f37b1fa0fa3 start_thread

/usr/local/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py:229: SessionNotCreatedException
------------------------------ Captured log setup ------------------------------
DEBUG    selenium.webdriver.common.selenium_manager:selenium_manager.py:101 Selenium Manager binary found at: /usr/local/lib/python3.8/site-packages/selenium/webdriver/common/linux/selenium-manager
DEBUG    selenium.webdriver.common.selenium_manager:selenium_manager.py:114 Executing process: /usr/local/lib/python3.8/site-packages/selenium/webdriver/common/linux/selenium-manager --browser chrome --language-binding python --output json
DEBUG    selenium.webdriver.common.selenium_manager:selenium_manager.py:140 Driver path: /root/.cache/selenium/chromedriver/linux64/130.0.6723.116/chromedriver
DEBUG    selenium.webdriver.common.selenium_manager:selenium_manager.py:140 Browser path: /root/.cache/selenium/chrome/linux64/130.0.6723.116/chrome
DEBUG    selenium.webdriver.common.service:service.py:225 Started executable: `/root/.cache/selenium/chromedriver/linux64/130.0.6723.116/chromedriver` in a child process with pid: 3204 using 0 to output -3
DEBUG    selenium.webdriver.remote.remote_connection:remote_connection.py:390 POST http://localhost:33777/session {'capabilities': {'firstMatch': [{}], 'alwaysMatch': {'browserName': 'chrome', 'pageLoadStrategy': <PageLoadStrategy.normal: 'normal'>, 'browserVersion': None, 'goog:chromeOptions': {'extensions': [], 'binary': '/root/.cache/selenium/chrome/linux64/130.0.6723.116/chrome', 'args': ['start-maximized', 'enable-automation', '--headless', '--no-sandbox', '--disable-dev-shm-usage', '--disable-browser-side-navigation', '--disable-gpu']}}}}
DEBUG    urllib3.connectionpool:connectionpool.py:243 Starting new HTTP connection (1): localhost:33777
DEBUG    urllib3.connectionpool:connectionpool.py:546 http://localhost:33777 "POST /session HTTP/11" 500 0
DEBUG    selenium.webdriver.remote.remote_connection:remote_connection.py:423 Remote response: status=500 | data={"value":{"error":"session not created","message":"session not created: Chrome failed to start: exited normally.\n  (session not created: DevToolsActivePort file doesn't exist)\n  (The process started from chrome location /root/.cache/selenium/chrome/linux64/130.0.6723.116/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)","stacktrace":"#0 0x55af503386da \u003Cunknown>\n#1 0x55af4fe4cf80 \u003Cunknown>\n#2 0x55af4fe82db1 \u003Cunknown>\n#3 0x55af4fe7e6c5 \u003Cunknown>\n#4 0x55af4fec9cf9 \u003Cunknown>\n#5 0x55af4fec9346 \u003Cunknown>\n#6 0x55af4febd953 \u003Cunknown>\n#7 0x55af4fe8c72e \u003Cunknown>\n#8 0x55af4fe8d79e \u003Cunknown>\n#9 0x55af50303f1b \u003Cunknown>\n#10 0x55af50307eb8 \u003Cunknown>\n#11 0x55af502f142c \u003Cunknown>\n#12 0x55af50308a37 \u003Cunknown>\n#13 0x55af502d5fef \u003Cunknown>\n#14 0x55af50326ad8 \u003Cunknown>\n#15 0x55af50326ca0 \u003Cunknown>\n#16 0x55af50337556 \u003Cunknown>\n#17 0x7f37b1fa0fa3 start_thread\n"}} | headers=HTTPHeaderDict({'Content-Length': '993', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG    selenium.webdriver.remote.remote_connection:remote_connection.py:452 Finished Request

@pytest.fixture
    def browser():
        '''
        This fixture will initialize the webdriver instance
        and will close the driver after the execution of tests.
        '''
        options = Options()
        options.add_argument("start-maximized")
        options.add_argument("enable-automation")
        options.add_argument("--headless")
        options.add_argument("--no-sandbox")
        options.add_argument("--disable-dev-shm-usage")
        options.add_argument("--disable-browser-side-navigation")
        options.add_argument("--disable-gpu")
    
        # Initialize ChromeDriver instance
>       driver = webdriver.Chrome(options=options)

My python webdriver configuration is as follows:-

@pytest.fixture
def browser():
    '''
    This fixture will initialize the webdriver instance 
    and will close the driver after the execution of tests. 
    '''
    options = Options()
    options.add_argument("start-maximized")
    options.add_argument("enable-automation")
    options.add_argument("--headless")
    options.add_argument("--no-sandbox")
    options.add_argument("--disable-dev-shm-usage")
    options.add_argument("--disable-browser-side-navigation")
    options.add_argument("--disable-gpu")

    # Initialize ChromeDriver instance
    driver = webdriver.Chrome(options=options)
    driver.implicitly_wait(5) #sec
    
    yield driver
    driver.quit()

How can I fix the above issue? I would be really happy if somebody gives me the solution to fix this error and run integration tests in Jenkins.

A frequent cause of ‘Chrome failed to start’ is that the agent does not have a display device that Chrome can use. Chrome needs a Windows desktop session or a Linux display session. Is your agent configured to provide Chrome a place to display itself?