My Clair pod is restarting at regular intervals

7/20/2021

I am using Clair for Vulnerability checks in my harbor.

Services like Clair that have thousands of hosts continually hitting the hosting git server(https://git.launchpad.net/ubuntu-cve-tracker/) saturate the server, and so there are scaling measures in place that causes it to return a 503 error when too many clients are concurrently hitting it.

These are my errors in my Clair pod:

{"Event":"could not pull ubuntu-cve-tracker repository","Level":"error","Location":"ubuntu.go:174",
"Time":"2021-06-25 06:38:32.859806","error":"exit status 128",
"output":"Cloning into '.'...

fatal: unable to access '[https://git.launchpad.net/ubuntu-cve-tracker/':|https://git.launchpad.net/ubuntu-cve-tracker/%27:] 
The requested URL returned error: 503\n"} 

{"Event":"an error occured when fetching update","Level":"error","Location":"updater.go:246",
"Time":"2021-06-25 06:38:32.859934","error":"could not download requested resource","updater name":"ubuntu"}
panic: runtime error: slice bounds out of range goroutine 549 [running]: github.com/coreos/clair/ext/vulnsrc/rhel.toFeatureVersions(0xc000208390, 0x2, 0xc000246070, 0x1, 0x1, 0xc0001bc200, 0x1, 0x1, 0x0, 0x908f38, ...) /go/src/github.com/coreos/clair/ext/vulnsrc/rhel/rhel.go:292 +0xc3b github.com/coreos/clair/ext/vulnsrc/rhel.parseRHSA(0x7fcc0f4a24b0, 0xc00038c0f0, 0xc00038c0f0, 0x7fcc0f4a24b0, 0xc00038c0f0, 0x8e2708, 0x4) /go/src/github.com/coreos/clair/ext/vulnsrc/rhel/rhel.go:182 +0x1c8 

As per https://bugs.launchpad.net/ubuntu-cve-tracker/+bug/1925337 this is a bug from the git server, and in that post, they are suggesting to get Clair to pull data from other sources instead which means an offline approach. So apart from the offline approach, is there any other way to decrease the number of hits to the git server for Vulnerability checks?

I have tried to control the number of hits to the git server, but nowhere have I found the configuration in Clair.

Does anyone have any idea how we can control the hits for Vulnerability checks or avoid restarts of my pod?

Also, I found schedule a scan(hourly, daily, or weekly) on my harbor UI, But how does scheduling the scan to say daily help?
Is it only at that point it will try to do the git clone to get the latest CVEs?

-- Anvesh Muppeda
clair
git
harbor
kubernetes

1 Answer

7/20/2021

Check first if this is linked to goharbor/harbor issue 14720: "clair restarts repeatedly when there is some issue with vulnerability repos", with logs like

{"Event":"Start fetching vulnerabilities","Level":"info","Location":"ubuntu.go:85","Time":"2021-04-21 19:18:24.446743","package":"Ubuntu"}
...
{"Event":"could not pull ubuntu-cve-tracker repository","Level":"error","Location":"ubuntu.go:174","Time":"2021-04-21 19:18:25.147515","error":"exit status 128","output":"Cloning into '.'...\nfatal: unable to access 'https://git.launchpad.net/ubuntu-cve-tracker/': The requested URL returned error: 503\n"}
{"Event":"an error occured when fetching update","Level":"error","Location":"updater.go:246","Time":"2021-04-21 19:18:25.147607","error":"could not download requested resource","updater name":"ubuntu"}
...
panic: runtime error: slice bounds out of range [25:24]

goroutine 327 [running]:
github.com/quay/clair/v2/ext/vulnsrc/rhel.toFeatureVersions(0xc0065215a8, 0x2, 0xc0000b4f08, 0x1, 0x1, 0xc006ef7aa0, 0x1, 0x1, 0x2, 0xc0000b4ef0, ...)
	/go/src/github.com/quay/clair/ext/vulnsrc/rhel/rhel.go:276 +0xbf8

It refers to quay/clair issue 1249, but the harbor case is closed with PR 15032, using CLAIRVERSION=v2.1.7

-- VonC
Source: StackOverflow