I find the defaults in the helm chart incredibly frustrating. So my suggestion is to do all plugin management yourself, inside your container, and clear the plugin dir on startup.
You can just include the ones in the plugin - helm-charts/values.yaml at aff05875d84a7d79f010abc8756a11940804131b · jenkinsci/helm-charts · GitHub - in your plugins.txt. I don’t think the helm chart actually uses anything natively other than casc (which you already have enabled).