Is it possible not to trigger automatic build from Git if only Jenkinsfile is being modified?

SCM Skip maybe? Where you can have a commit message skip the build.

I think you should treat your entire pipeline as development code. So you know if a change breaks ci for any reason early.