Concourse V7 - Unique Version History & Keyval Updates

March 20, 2023

Concourse V7 drops support for unique_version_history, it seems this property was used in conjunction with global resources (you can read about them here). See this excerpt from that document about them:

The basic concept of global resources is to share detected resource versions between all resources that have the same resource.type and resource.source configuration. Before v5.0.0, each pipeline resource had its own version history, associated to the resource by name. This meant that multiple pipelines with the same resource configs would redundantly collect the same version and metadata information.

Not exactly sure which Concourse version it was, but global resources were enabled by default, I’m assuming it was V6. In Concourse V7 unique_version_history is deprecated. Resources which previously didn’t share resource versions will when unique_version_history is removed from the resource type. This is not ideal for resources like time and keyval (which many pipelines seem to use.)

This issue explains global resources a little bit and mentions how unique_version_history functions.

Take this resource type for example:

resource_types:
  - name: keyval
    type: registry-image
    source:
      repository: <link to repo>
      tag: release-v1.0.6
    unique_version_history: true

unique_version_history: true <— This indicates versions stored in this resource should be unique to this pipeline only and should not be globally collected. However, this needs to be removed for V7. The keyval resource is usually used to save state between jobs without having to configure an external resource like S3 or other data stores. In the same pipeline are usually resources that look like this:

resources:
  - name: kv_deploy_development
    type: keyval

  - name: kv_deploy_main
    type: keyval

Because of the way global resources work, when unique_version_history is removed, when you check these resources you’ll see versions collected from all your other pipelines.

The fix that I’ve come up with for this is to update your resources from this:

resources:
  - name: kv_deploy_development
    type: keyval

  - name: kv_deploy_main
    type: keyval

To this:

resources:
  - name: kv_deploy_development
    type: keyval
    icon: file
    # This source configuration is not used by the keyval resource and is
    # arbitrary. Adding this source makes it unique within concourse and only
    # versions from this pipeline will be displayed. This is required due
    # to how global resources function in concourse. Previously, the
    # unique_version_history field could be included on the resource type
    # to prevent this issue, but that has been deprecated in concourse v7.
    source:
      project: ((project_name))-development

  - name: kv_deploy_main
    type: keyval
    icon: file
    # This source configuration is not used by the keyval resource and is
    # arbitrary. Adding this source makes it unique within concourse and only
    # versions from this pipeline will be displayed. This is required due
    # to how global resources function in concourse. Previously, the
    # unique_version_history field could be included on the resource type
    # to prevent this issue, but that has been deprecated in concourse v7.
    source:
      project: ((project_name))-main

Then remove unique_version_history from the resource type:

resource_types:
  - name: keyval
    type: registry-image
    source:
      repository: <link to repo>
      tag: release-v1.0.6

The reason this works is because resources are essentially a black box to Concourse. We add this source to each resource to make them unique in the eyes of Concourse, which stops all the versions from being collected into one resource and simulates what unique_version_history used to do. They have no effect on the actual resource’s implementation.

source:
    project: ((project_name))-main

The project key and value are arbitrary and needs to be unique between all the different pipelines & resources you have deployed. I contemplated using a GUID, but settled on just prefixing the value with the project name which should make everything unique. It’s not the most elegant solution, but I haven’t seen or come up with any other solutions to fix it. After implementing this, you should see only versions generated by the pipeline.

I have noticed that if you remove unqiue_version_history without making the changes outlined above, the pipeline does work still. However, I’ve noticed issues with pipelines, although it wasn’t 100% confirmed that it was because of unique_version_history being removed. At the very least, removing unique_version_history causes your keyval resources to become cluttered with versions that aren’t used by your pipeline. If you want to pin a specific version it makes it hard to find because all the different versions generated by your other pipelines are shown within the resource.

The best way to ensure your pipelines work as they did previously is to add the unique source to each keyval resource.