download advisory database

[javixeneize]

Hi

Can the advisory database be downloaded? I would prefer to have a local database and query my data against it instead of calling the graphql endpoint for advisories for each dependency

Thanks

[KateCatlin]

Hi @javixeneize!
Interesting question. We did not talk about that as a feature but will note this down as a data point for consideration.

I suppose for now you could git clone the repository and download it, just keep in mind our license terms here: https://docs.github.com/en/github/site-policy/github-terms-for-additional-products-and-features#12-advisory-database

[javixeneize]

Thanks. Yeah, cloning it was an option but I was more wondering if you generate a daily db file as nvd does for example. This would be useful

regarding the license, the bottom line is than I’m planning to build an open source sca tool that use ghas as the database. It does not necessarily need to scan code hosted in GitHub, it would be a generic tool that would scan any dependency in any application. Would this be a valid use case for this license?

thanks

[KateCatlin]

Hey @javixeneize, generally with the CC-BY license any usage is ok as long as it's attributed back to us. But I can't answer specific legal questions so I'll have to refer you back to the license itself!

Sounds like what you're building could be really cool!

[G-Rath]

@KateCatlin I've actually just finished an initial build of a detector tool powered by this database (with native parsers for some of the ecosystem lockfiles and a raw CSV based input for manual lookups), so am very interested in this too 😅

Originally it was just a "hack something together, see if it flags something our current scanners don't" type project, but frankly it's working very well + the database seems super stable (I had concerns it might have e.g. edge cases with version ranges that'd make it painful to maintain our own tool, but so far no issues 🤷‍♂️) and I could see it being very useful to have an offline tool that lets you throw a package + version and know if it has any vulnerabilities so am going to open-source it.

I've not really dealt with licensing too heavily in the past, so was wondering if someone from GitHub wouldn't mind looking over it once I push it up? (Which'll probably be tomorrow - currently sick in bed)

(I understand this is squarely in Legal space, so I understand if you have to say no, but figured no harm in asking 🙂)

[javixeneize]

Sounds cool. Let me know if I can help with that. For me, GitHub advisories are a reliable source and much better structured than nvd database, so I won’t be surprised if this becomes the main source in the industry in the short term.
I’m just wary about the rate limit when calling the API, that’s why I think the local database would be a great idea. You just pull the advisories and that’s it, you don’t need to query the api anymore

[G-Rath]

@javixeneize thanks, definitely welcome to help! Right now I'm in the middle of sort of porting it over from TypeScript to Go, and implementing some more stuff that I've got the design for but just need to write the code.

I agree with what you're said in full - related to that too is why I've been converting it to Go as I think it'd be even better having a single fast "point and scan" type binary.

I'll try and get it up into a repo today - it's probably going to be rough (mainly in documentation) since I hit a bit of a bump with being sick this weekend, but I should be able to have most of that finished by next week (I'd push it right now, but after seeing this I want to make sure I've at least got a basic attribution to this repo).

[G-Rath]

Ok so I went ahead and pushed the code up: https://github.com/G-Rath/osv-detector 🎉

I've not done any releases or anything yet because there's still a few core features I'm got locally to finish off and push up, which I probably won't have time for until the weekend; I also found that the version comparison packages available are generally all wanting SemVer only so am going to write a custom parser for that (and this means the detector doesn't work well on some ecosystems like Ruby; though npm and packagist should be fine).

For now I've got it under my name, but I'm considering moving it to be under my works org (Ackama) as they're a big support of this sort of stuff and could help get more love and attention if it proves useful.

[G-Rath]

@KateCatlin @javixeneize I've just published the first release: https://github.com/G-Rath/osv-detector/releases/tag/v0.1.0 - it should be stable, so am hoping to have a v1 release soon (I just want to land some more core features first)