403 forbidden with service account from GCP

Using Jenkins
1

What I am trying to do:

  • A Jenkins Pipeline that is connected to a Github repo with webhooks.
  • There is a Terraform file on GitHub for what resource will be deployed to a GCP project
  • When Jenkins detected code changed on Github, it will trigger a build.

What problem I encountered:

  • I am trying to deploy a bucket by this pipeline but facing 403 forbidden issue
  • The service account being used is granted permission (owner)
  • the error message from jenkins pipeline:
+ terraform init -backend-config=bucket=terraform-state-bucket-project-data-sharing -migrate-state
e[0me[1mInitializing the backend...e[0m
e[31m╷e[0me[0m
e[31m│e[0m e[0me[1me[31mError: e[0me[0me[1mError loading state: writing "gs://terraform-state-bucket-project-data-sharing/terraform/state/default.tflock" failed: googleapi: Error 403: Access denied., forbidden
e[31m│e[0m e[0mstorage: object doesn't existe[0m
e[31m│e[0m e[0m
e[31m│e[0m e[0me[0m
e[31m╵e[0me[0m

What I have tried:

  • I tried to use the same service account in the cloud shell to create bucket, the account works:
  • I also tried to grant all possible permission to the account:

Please provide suggestions, thanks!

here is my code: GitHub - ShihWen/data-sharing: gap-tf-jenkins
More detail for the issue: google cloud platform - Integration between Jenkins and GCP service account, 403 forbidden - Stack Overflow
Jenkins setup:

  • version: 2.492.3
  • Plugins:
ant: 513.vde9e7b_a_0da_0f
antisamy-markup-formatter: 173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api: 4.5.14-269.vfa_2321039a_83
asm-api: 9.8-135.vb_2239d08ee90
bootstrap5-api: 5.3.3-2
bouncycastle-api: 2.30.1.80-256.vf98926042a_9b_
branch-api: 2.1214.v3f652804588d
build-timeout: 1.37
caffeine-api: 3.2.0-166.v72a_6d74b_870f
checks-api: 367.v18b_7f530e54a_
cloudbees-folder: 6.999.v42253c105443
command-launcher: 118.v72741845c17a_
commons-lang3-api: 3.17.0-87.v5cf526e63b_8b_
commons-text-api: 1.13.0-153.v91dcd89e2a_22
credentials-binding: 687.v619cb_15e923f
credentials: 1413.va_51c53703df1
dark-theme: 524.vd675b_22b_30cb_
display-url-api: 2.209.v582ed814ff2f
durable-task: 587.v84b_877235b_45
echarts-api: 5.6.0-2
eddsa-api: 0.3.0.1-19.vc432d923e5ee
email-ext: 1876.v28d8d38315b_d
font-awesome-api: 6.7.2-1
gcloud-sdk: 0.0.3
gcp-java-sdk-storage: 26.23.0-31.va_efec42c4f8c
git-client: 6.1.2
git: 5.7.0
github-api: 1.321-488.v9b_c0da_9533f8
github-branch-source: 1815.v9152b_2ff7a_1b_
github: 1.43.0
google-oauth-plugin: 1.335.ve6de40e2db_18
gradle: 2.14.1
gson-api: 2.12.1-113.v347686d6729f
instance-identity: 203.v15e81a_1b_7a_38
ionicons-api: 82.v0597178874e1
jackson2-api: 2.18.3-402.v74c4eb_f122b_2
jakarta-activation-api: 2.1.3-2
jakarta-mail-api: 2.1.3-2
javax-activation-api: 1.2.0-8
javax-mail-api: 1.6.2-11
jaxb: 2.3.9-133.vb_ec76a_73f706
jdk-tool: 83.v417146707a_3d
jjwt-api: 0.11.5-120.v0268cf544b_89
joda-time-api: 2.14.0-127.v7d9da_295a_d51
jquery3-api: 3.7.1-3
json-api: 20250107-125.v28b_a_ffa_eb_f01
json-path-api: 2.9.0-148.v22a_7ffe323ce
junit: 1319.v000471ca_e5e2
ldap: 780.vcb_33c9a_e4332
mailer: 489.vd4b_25144138f
matrix-auth: 3.2.6
matrix-project: 847.v88a_f90ff9f20
metrics: 4.2.21-464.vc9fa_a_0d6265d
mina-sshd-api-common: 2.15.0-161.vb_200831a_c15b_
mina-sshd-api-core: 2.15.0-161.vb_200831a_c15b_
oauth-credentials: 0.657.v7d8dd90b_0382
okhttp-api: 4.11.0-189.v976fa_d3379d6
pam-auth: 1.12
pipeline-build-step: 557.v95d96f77b_2b_8
pipeline-gcp: 22.vcc10eff7c13f
pipeline-github-lib: 65.v203688e7727e
pipeline-graph-analysis: 235.vb_a_a_36b_f248c2
pipeline-graph-view: 423.v765c49ca_da_3f
pipeline-groovy-lib: 752.vdddedf804e72
pipeline-input-step: 517.vf8e782ee645c
pipeline-milestone-step: 127.vb_52887ca_3b_6d
pipeline-model-api: 2.2247.va_423189a_7dff
pipeline-model-definition: 2.2247.va_423189a_7dff
pipeline-model-extensions: 2.2247.va_423189a_7dff
pipeline-rest-api: 2.37
pipeline-stage-step: 322.vecffa_99f371c
pipeline-stage-tags-metadata: 2.2247.va_423189a_7dff
pipeline-stage-view: 2.37
plain-credentials: 195.vb_906e9073dee
plugin-util-api: 6.1.0
resource-disposer: 0.25
saferestart: 102.v4dc1b_9636a_ee
scm-api: 704.v3ce5c542825a_
script-security: 1373.vb_b_4a_a_c26fa_00
snakeyaml-api: 2.3-125.v4d77857a_b_402
ssh-credentials: 355.v9b_e5b_cde5003
ssh-slaves: 3.1031.v72c6b_883b_869
sshd: 3.353.v2b_d33c46e970
structs: 343.vdcf37b_a_c81d5
terraform: 1.0.10
theme-manager: 278.v2e3c063e42cc
timestamper: 1.28
token-macro: 444.v52de7e9c573d
trilead-api: 2.192.vc50a_d147e369
variant: 70.va_d9f17f859e0
workflow-aggregator: 608.v67378e9d3db_1
workflow-api: 1366.vf1fb_e1a_f6b_22
workflow-basic-steps: 1079.vce64b_a_929c5a_
workflow-cps: 4080.va_15b_44a_91525
workflow-durable-task-step: 1405.v1fcd4a_d00096
workflow-job: 1508.v9cb_c3a_a_89dfd
workflow-multibranch: 803.v08103b_87c280
workflow-scm-step: 437.v05a_f66b_e5ef8
workflow-step-api: 700.v6e45cb_a_5a_a_21
workflow-support: 963.va_600813d04a_a_
ws-cleanup: 0.48
2

From where does the terraform init execution gets the credentials?
You have a withCredentials in the first stage but not where the terraform init is done

3

Hi @mawinter69 ,
I am quite new to Jenkins world,
how would you suggest to amend the pipeline code in my Terraform init stage?

4

With

                withCredentials([file(credentialsId: 'gcp-sa-dev', variable: 'GOOGLE_APPLICATION_CREDENTIALS')]) { // Ensure 'gcp-sa-dev' is your Credential ID in Jenkins
                    sh 'gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS'
                    sh 'gcloud config set project $GCP_PROJECT_ID'
                    echo "GCP Authentication Configured as: ${SERVICE_ACCOUNT_EMAIL}"

                    // sh 'terraform init -backend-config="bucket=${TF_STATE_BUCKET}" -migrate-state'
                    // sh 'terraform validate'
                    // sh 'terraform plan -out=tfplan'
                    // archiveArtifacts artifacts: 'tfplan'
                    // input message: 'Approve Terraform Apply to Production?', ok: 'Proceed with Apply'
                    // sh 'terraform apply tfplan'
                }

You create a temporary file that is use as parameter --key-file for gcloud auth. Once this block is finished the temporary file is deleted. So in your later steps will not be able to use that file outside of the block.
What you would need to do it copy this file to another file, use this in the call to gcloud and add an always condition to the post where this file is deleted.

5

You can also try the following:

pipeline {
    agent any
    tools {
        terraform 'Terraform-v1.11.3' // Ensure tool name matches your Jenkins Global Tool Configuration
    }
    environment {
        // --- REPLACE THESE WITH YOUR ACTUAL VALUES ---
        GCP_PROJECT_ID = 'open-data-v2-cicd'
        TF_STATE_BUCKET = 'terraform-state-bucket-project-data-sharing'
        GCP_REGION = 'asia-east1'
        SERVICE_ACCOUNT_EMAIL = 'jenkins-tf-dev@open-data-v2-cicd.iam.gserviceaccount.com'
        GOOGLE_APPLICATION_CREDENTIALS = credentials('gcp-sa-dev')
   }
...

Then you should have the credential in all stages automatically

1 Like
6

Works like magic, thanks @mawinter69 !