Missing openshift api-resources

[dan-m8t]

Hey,

I am playing around with datree but fail to get a working validation with specific OpenShift api resources.
I exported the resources via the crd extractor tool but datree just throws an error:

helm template ~/work/mychart/ -f ~/work/mychart/values/myproject/values.yaml | datree test - --schema-location '$HOME/.datree/crdSchemas/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json'
>>  File: /tmp/datree_temp_362123854.yaml

[V] YAML validation
[X] Kubernetes schema validation

❌  k8s schema validation error: could not find schema for Group

❌  k8s schema validation error: could not find schema for Group

To troubleshoot: refer to our docs [https://hub.datree.io/troubleshooting#schema-validation-failure]
To ignore this failure: use the CLI flag `--ignore-missing-schemas`

[?] Policy check didn't run for this file


(Summary)

- Passing YAML validation: 1/1

- Passing Kubernetes (1.20.0) schema validation: 0/1

- Passing policy check: 0/1

+-----------------------------------+------------------------------------------------------+
| Enabled rules in policy "Default" | 21                                                   |
| Configs tested against policy     | 0                                                    |
| Total rules evaluated             | 0                                                    |
| Total rules skipped               | 0                                                    |
| Total rules failed                | 0                                                    |
| Total rules passed                | 0                                                    |
| See all rules in policy           | https://app.datree.io/login?t=xxxx |
+-----------------------------------+------------------------------------------------------+

I double checked what the crd extractor extracted and the group json schema is part of it:

{
  "description": "Group represents a referenceable set of Users",
  "type": "object",
  "required": [
    "users"
  ],
  "properties": {
    "apiVersion": {
      "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources",
      "type": "string"
    },
    "kind": {
      "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds",
      "type": "string"
    },
    "metadata": {
      "$ref": "_definitions.json#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta"
    },
    "users": {
      "description": "Users is the list of users in this group.",
      "type": "array",
      "items": {
        "type": "string"
      }
    }
  },
  "x-kubernetes-group-version-kind": [
    {
      "group": "user.openshift.io",
      "kind": "Group",
      "version": "v1"
    }
  ],
  "$schema": "http://json-schema.org/schema#"
}

Is there something wrong with the schema?

Thanks

[eyarz]

@dan-m8t the CRD-extractor is extracting only CRDs from your cluster.

It looks like you're trying to validate an OpenShift native resource ("kind": "Group"), so you will need to use the --schema-location flag differently, depending on your OpenShift version.

What is your OpenShift version?

[dan-m8t]

Thanks for clarifying @eyarz

I use OpenShift 4.8 and 4.10.

What do I need to do?

Thanks

[eyarz]

You have two locations:

• OpenShift v4.1.0: https://github.com/garethr/openshift-json-schema/tree/master/v4.1.0-standalone
• OpenShift v4.8.2: https://github.com/datreeio/CRDs-catalog/tree/openshift/openshift/v4.8.2-standalone

Both locations work with the same schema naming convention: [RAW_URL]/{{.ResourceKind}}{{.KindSuffix}}.json

Example (OpenShift v4.1.0):

datree test [MANIFEST] --schema-location 'https://raw.githubusercontent.com/garethr/openshift-json-schema/master/v4.1.0-standalone/{{.ResourceKind}}{{.KindSuffix}}.json'

Example (OpenShift v4.8.2):

datree test [MANIFEST] --schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/openshift/openshift/v4.8.2-standalone/{{.ResourceKind }}{{.KindSuffix}}.json'

You can also read here about how you can pull all the schemas from your cluster here:
https://cloud.redhat.com/blog/validating-openshift-manifests-in-a-gitops-world
(use datree instead of kubeval which is not maintained anymore)

[dan-m8t]

Thanks for pointing me into the right direction but I still get the same error about the Group schema :(

helm template ~/work/chart/ -f ~/work/chart/values/project/values.yaml | datree test -  --no-record --schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/openshift/openshift/v4.8.2-standalone/{{.ResourceKind }}{{.KindSuffix}}.json'
>>  File: /tmp/datree_temp_2894782400.yaml

[V] YAML validation
[X] Kubernetes schema validation

❌  k8s schema validation error: could not find schema for Group

❌  k8s schema validation error: could not find schema for Group`

I'll try do clone the OpenShift schema and test locally

[dan-m8t]

Even this is not working, manifest is just a group, checked against the particular json schema:

datree test group1.yaml --no-record --schema-location 'CRDs-catalog/openshift/openshift/v4.8.2-standalone/group-user-v1.json'                                                                                                                                                     INT ✘  
>>  File: /home/addanielm/work/helm-validate/group1.yaml

[V] YAML validation
[X] Kubernetes schema validation

❌  k8s schema validation error: could not find schema for Group

Heres the manifest I want to check:

apiVersion: user.openshift.io/v1
kind: Group
metadata:
  labels:
    app.kubernetes.io/instance: some-project-defaults
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: project
    app.kubernetes.io/version: 0.1.0
    helm.sh/chart: okd-project-0.1.0
    doesntmatter/argocd-devops: some-managed-argocd-field
    company/name: someproject
  name: project-admins-group
users:
- someuser

I had the thought that the managed fiels/labels may cause an issue? This is an Argo CD managed Helm Chart which injects those.

But it even fails with a manual created manifest:

apiVersion: user.openshift.io/v1
kind: Group
metadata:
  name: doesntmatter
users:
  - someuser

datree output:

datree test group2.yaml --no-record --schema-location 'CRDs-catalog/openshift/openshift/v4.8.2-standalone/group-user-v1.json'                                                                                                                                                            ✔  
>>  File: /home/addanielm/work/helm-validate/group2.yaml

[V] YAML validation
[X] Kubernetes schema validation

❌  k8s schema validation error: could not find schema for Group

[dan-m8t]

Same issue for other OpenShift specific resources:

datree test routetest.yaml --no-record --schema-location 'CRDs-catalog/openshift/openshift/v4.8.2-standalone/route-route-v1.json'                                                                                                                                              2 ✘  6s   
>>  File: /home/addanielm/work/helm-validate/routetest.yaml

[V] YAML validation
[X] Kubernetes schema validation

❌  k8s schema validation error: could not find schema for Route

I'll take a look at getting those schemas for my cluster locally.

[dan-m8t]

And also same for extracted api resources as in the mentioned RedHat article, not sure if I do something wrong here.

[eyarz]

@dan-m8t when I'm testing your kind: Group manifest with the OpenShift v4.1.0 schema, it is passing.
I never tested the OpenShift v4.8.2 schemas (which is why they are still on a side branch), but I extracted them from an OpenShift cluster, as mentioned in the RedHat guide I provided.

I suspect the problem is with the v4.8.2 schemas themselves.
I'm not an OpenShift expert or even have access to an OpenShift cluster, so I will need your help to solve this one :)
Can you provide another manifest that uses a native OpenShift resource?
This way, we can verify if this is the problem.

[dan-m8t]

Sorry, was busy.
For clearance: I use OpenShift 4.10 and 4.8, not 4.1 ;)

Here's another OpenShift native resource which fails:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: route-example
spec:
  host: www.example.com
  path: "/"
  to:
    kind: Service
    name: service-name

I'm glad to help, if you need something OpenShift related tell me :)

[eyarz]

Thank you for the help!
I see I'm getting the same error with this manifest, so this brings me to the conclusion that the problem is indeed with the schemas I extracted from the OpenShift cluster, and not with the validation process.

Because I don't have access to this OpenShift cluster anymore, I don't know what went wrong when I pulled the schemas.
You can try and use the guide and pull the schemas from your OpenShift cluster.

[dan-m8t]

I'll give that a shot and report back to you, thanks.

[dan-m8t]

No luck here, I tried to extract the schemas with both ways described in the blog post. Does not work :(.

Would you like to try the route example with the schema I extracted? If so, try this:

{
  "description": "A route allows developers to expose services through an HTTP(S) aware load balancing and proxy layer via a public DNS entry. The route may further specify TLS options and a certificate, or specify a public CNAME that the router should also accept for HTTP and HTTPS traffic. An administrator 
typically configures their router to be visible outside the cluster firewall, and may also add additional security, caching, or traffic controls on the service content. Routers usually talk directly to the service endpoints.\n\nOnce a route is created, the `host` field may not be changed. Generally, routers use the oldest route with a given host when resolving conflicts.\n\nRouters are subject to additional customization and may support additional controls via the annotations field.\n\nBecause administrators may configure multiple routers, the route status field is used to return information to clients about the 
names and states of the route under each router. If a client chooses a duplicate name, for instance, the route status conditions are used to indicate the route cannot be chosen.\n\nTo enable HTTP/2 ALPN on a route it requires a custom (non-wildcard) certificate. This prevents connection coalescing by clients, notably web browsers. We do not support HTTP/2 ALPN on routes that use the default certificate because of the risk of connection re-use/coalescing. Routes that do not have their own custom certificate will not be HTTP/2 ALPN-enabled on either the frontend or the backend.",
  "type": "object",
  "required": [
    "spec"
  ],
  "properties": {
    "apiVersion": {
      "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources",
      "type": [
        "string",
        "null"
      ],
      "enum": [
        "v1",
        "route.openshift.io/v1"
      ]
    },
    "kind": {
      "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds",
      "type": [
        "string",
        "null"
      ],
      "enum": [
        "Route"
      ]
    },
    "metadata": {
      "$ref": "_definitions.json#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta"
    },
    "spec": {
      "description": "spec is the desired state of the route",
      "$ref": "_definitions.json#/definitions/com.github.openshift.api.route.v1.RouteSpec"
    },
    "status": {
      "description": "status is the current state of the route",
      "$ref": "_definitions.json#/definitions/com.github.openshift.api.route.v1.RouteStatus"
    }
  },
  "x-kubernetes-group-version-kind": [
    {
      "group": "",
      "kind": "Route",
      "version": "v1"
    },
    {
      "group": "route.openshift.io",
      "kind": "Route",
      "version": "v1"
    }
  ],
  "$schema": "http://json-schema.org/schema#"
}

[dol]

Try this steps:

mkdir tmp
cd tmp
wget https://github.com/openshift/api/archive/refs/heads/release-4.10.zip
unzip release-4.10.zip
wget https://raw.githubusercontent.com/yannh/kubeconform/master/scripts/openapi2jsonschema.py
mkdir schema
cd schema
# Depending on you ** might not work. In zsh it's working.
python3 ../openapi2jsonschema.py ../api-release-4.10/**/*.crd.yaml
# Output is the file to check
# cp ../../../output.yaml .
datree test --schema-location $PWD/'{{ .ResourceKind }}_{{ .ResourceAPIVersion }}.json' output.yaml

[dan-m8t]

That seems to lead me into the right direction.

To get all the resources I needed to do this:

python3 ../openapi2jsonschema.py ../api-release-4.10/**/**/*.crd.yaml

With your command it was missing for example the route schema under route/v1

Trying to validate my example Route from above this works:

datree test --schema-location $PWD/'{{ .ResourceKind }}_{{ .ResourceAPIVersion }}.json' ../../route.yaml

(Summary)

- Passing YAML validation: 1/1

- Passing Kubernetes (1.20.0) schema validation: 1/1

- Passing policy check: 1/1

+-----------------------------------+------------------------------------------------------+
| Enabled rules in policy "Default" | 21                                                   |
| Configs tested against policy     | 1                                                    |
| Total rules evaluated             | 21                                                   |
| Total rules skipped               | 0                                                    |
| Total rules failed                | 0                                                    |
| Total rules passed                | 21                                                   |

I am not sure why there are missing schemas for specific OpenShift resources like Group in that OpenShift repo, maybe I'll find it somewhere.

Thank you, I'll report back if I make more progress.

[tricktron]

@dan-m8t I created a pull request, which adds the openshift schemas for v4.11. You should be able to validate your openshift route with the following command:

datree test --schema-version "1.24.6" --schema-location https://raw.githubusercontent.com/tricktron/CRDs-catalog/f-openshift-v4.11/openshift.io/v4.11/'{{ .ResourceKind }}_{{ .ResourceAPIVersion }}.json' --no-record route.yaml

[dan-m8t]

Thank you very much @tricktron - I'll give it a shot tomorrow!