Applegate Blog

Concourse V7 - Unique Version History & Keyval Updates

March 20, 2023

Concourse version 7 drops support for unique_version_history, it seems this property was used in conjunction with global resources (you can read about them in more detail here). See this excerpt from the Concourse docs:

The basic concept of global resources is to share detected resource versions between all resources that have the same resource.type and resource.source configuration. Before v5.0.0, each pipeline resource had its own version history, associated to the resource by name. This meant that multiple pipelines with the same resource configs would redundantly collect the same version and metadata information.

It seems in version 6 of Concourse that global resources were enabled by default. In Concourse version 7 unique_version_history is deprecated. Resources which previously did not share resource versions now will when unique_version_history is removed from the resource type. This is not ideal for resources like time and keyval (which many pipelines use.)

This issue explains global resources a little bit and mentions how unique_version_history functions.

Take this resource type for example:

resource_types:
  - name: keyval
    type: registry-image
    source:
      repository: <link to repo>
      tag: release-v1.0.6
    unique_version_history: true

unique_version_history: true <— This indicates versions stored in this resource should be unique to this pipeline only and should not be globally collected. However, this needs to be removed in version 7 of Concourse. The keyval resource is usually used to save state between jobs without having to configure an external resource like S3 or other data store. In the same pipeline there are usually resources that look like this:

resources:
  - name: deploy_development_version
    type: keyval

  - name: deploy_main_version
    type: keyval

Because of the way global resources work, when unique_version_history is removed, when you check these resources you’ll see versions collected from all your other pipelines.

The fix that I’ve come up with for this is to update your resources from this:

resources:
  - name: deploy_development_version
    type: keyval

  - name: deploy_main_version
    type: keyval

To this:

resources:
  - name: deploy_development_version
    type: keyval
    icon: file
    # This source configuration is not used by the keyval resource and is
    # arbitrary. Adding this source makes it unique within concourse and only
    # versions from this pipeline will be displayed. This is required due
    # to how global resources function in concourse. Previously, the
    # unique_version_history field could be included on the resource type
    # to prevent this issue, but that has been deprecated in concourse v7.
    source:
      project: ((project_name))-development

  - name: deploy_main_version
    type: keyval
    icon: file
    # This source configuration is not used by the keyval resource and is
    # arbitrary. Adding this source makes it unique within concourse and only
    # versions from this pipeline will be displayed. This is required due
    # to how global resources function in concourse. Previously, the
    # unique_version_history field could be included on the resource type
    # to prevent this issue, but that has been deprecated in concourse v7.
    source:
      project: ((project_name))-main

Then remove unique_version_history from the resource type:

resource_types:
  - name: keyval
    type: registry-image
    source:
      repository: <link to repo>
      tag: release-v1.0.6

The reason this works is because resources are essentially a black box to Concourse. We add this source to each resource to make them unique in the eyes of Concourse, which stops all the versions from being collected into one resource and simulates what unique_version_history used to do. They have no effect on the actual resource’s implementation.

source:
    project: ((project_name))-main

The project key and value are arbitrary and needs to be unique between all the different pipelines and resources you have deployed. I contemplated using a GUID, but settled on just prefixing the value with the project name which should make everything unique. It’s not the most elegant solution, but I haven’t seen or come up with any other solutions to fix it. After implementing this, you should see only versions generated by the pipeline.

I have noticed that if you remove unique_version_history without making the changes outlined above, the pipeline does work still. However, I’ve noticed issues with pipelines, although it wasn’t 100% confirmed that it was because of unique_version_history being removed. At the very least, removing unique_version_history causes your keyval resources to become cluttered with versions that aren’t used by your pipeline. If you want to pin a specific version it makes it hard to find because all the different versions generated by your other pipelines are shown within the resource.

The best way to ensure your pipelines work as they did previously is to add the unique source to each keyval resource.

© 2024