At the request of serviceops all racking tasks for new hardware also have a sub-task for service ops implementation tracking.
Once parent task T326366 shows resolved, this can proceed via the service ops team.
topic branch with related patches: https://gerrit.wikimedia.org/r/q/topic:gerrit-bullseye
| 1 | # schedule and announce downtime |
|---|---|
| 2 | # on gerrit1001: shortly before the scheduled downtime: |
| 3 | # on gerrit1001, as root, in a screen: rsync -avp --delete --bwlimit=100m /var/lib/gerrit2/review_site/ rsync://gerrit1003.wikimedia.org/gerrit-var-lib/ |
| 4 | # on gerrit1001, as root, in a screen: rsync -avp --delete --bwlimit=100m /srv/gerrit/ rsync://gerrit1003.wikimedia.org/gerrit-data/ |
| 5 | # on gerrit1003: rsync -avp /srv/gerrit/plugins/lfs/ /srv/gerrit/data/lfs/ |
| 6 | # on gerrit1003: chown -R gerrit2:gerrit2 /var/lib/gerrit2 |
| 7 | # on gerrit1003: chown -R gerrit2:gerrit2 /srv/gerrit |
| 8 | # scheduled downtime begins / IRC announcement |
| 9 | # on cumin1001:sudo cookbook sre.hosts.downtime -r 'maintenance' -D 30 gerrit1001.wikimedia.org |
| 10 | # on cumin1001:sudo cookbook sre.hosts.downtime -r 'maintenance' -H 1 gerrit1003.wikimedia.org |
| 11 | # on icinga.wikimedia.org - manually schedule downtime for the checks connected to virtual server "gerrit.wikimedia.org". The cookbook does not find this virtual host. |
| 12 | # on gerrit1003: disable puppet; stop gerrit? (sudo disable-puppet 'gerrit maintenance'; systemctl stop gerrit) |
| 13 | # merge DNS change that removes gerrit-new and switches IP of gerrit.wikimedia.org - in web UI of gerrit(-old) |
| 14 | # run authdns-update on ns0.wikimedia.org, see the diff but do NOT commit yet |
| 15 | # on gerrit1001: disable puppet; stop gerrit! (sudo disable-puppet 'gerrit maintenance'; systemctl stop gerrit) |
| 16 | # on gerrit1001, as root, in a screen: rsync -avp --delete --bwlimit=100m /var/lib/gerrit2/review_site/ rsync://gerrit1003.wikimedia.org/gerrit-var-lib/ |
| 17 | # on gerrit1001, as root, in a screen: rsync -avp --delete --bwlimit=100m /srv/gerrit/ rsync://gerrit1003.wikimedia.org/gerrit-data/ |
| 18 | # on gerrit1003: rsync -avp /srv/gerrit/plugins/lfs/ /srv/gerrit/data/lfs/ |
| 19 | # on gerrit1003: chown -R gerrit2:gerrit2 /var/lib/gerrit2 |
| 20 | # on gerrit1003: chown -R gerrit2:gerrit2 /srv/gerrit |
| 21 | # on gerrit1003: start gerrit |
| 22 | # say "yes" to authdns-update and actually merge DNS change that removes gerrit-new and switches IP of gerrit.wikimedia.org |
| 23 | # wait 5 minutes |
| 24 | # ..test https (https://gerrit.wikimedia.org in browser) |
| 25 | # ..test ssh (e.g. ssh [email protected] -p 29418) |
| 26 | # announce downtime is over |
| 27 | # ensure gerrit1001 has puppet disabled and/or services are masked |
| 28 | # grace period (how long?) |
| 29 | # decom old host -> https://phabricator.wikimedia.org/T336427 |
Related ObjectsStatus Subtype Assigned Task Unknown Object (Task) Resolved Papaul T326366 Q3:rack/setup/install gerrit1003 Open None T291916 Tracking task for Bullseye migrations in production Resolved Dzahn T327068 Bullseye upgrade for remaining Collab hosts Resolved Dzahn T326368 gerrit1003 service implementation task Resolved Dzahn T334524 add new and remove old Gerrit service IP from netbox and DNS Resolved Dzahn T336427 decom gerrit1001 Resolved Jclark-ctr T340077 decommission gerrit1001.wikimedia.org (dcops, netbox) Resolved BUG REPORT hashar T336524 IPv6 SSH to gerrit.wikimedia.org hangs (blackhole route?)
- Mentioned In
- T387833: Gerrit failover process
T338470: Rename gerrit2 unix user to gerrit and assign a fixed uid
T336524: IPv6 SSH to gerrit.wikimedia.org hangs (blackhole route?)
T335730: gerrit-replica seems out of date breaking extension distributor
T333143: Move Gerrit data out of root partition
T257317: scap deploy --init on deployment server fails on first puppet run
T334524: add new and remove old Gerrit service IP from netbox and DNS
T327068: Bullseye upgrade for remaining Collab hosts
T334521: upgrade gerrit servers to bullseye - Mentioned Here
- T336427: decom gerrit1001
T336339: Upgrade Gerrit from 3.5.5 to 3.5.6
P47782 gerrit migration plan/steps
T335730: gerrit-replica seems out of date breaking extension distributor
T333143: Move Gerrit data out of root partition
T257317: scap deploy --init on deployment server fails on first puppet run
T334521: upgrade gerrit servers to bullseye
T326366: Q3:rack/setup/install gerrit1003
Event Timeline
There are a very large number of changes, so older changes are hidden. Show Older ChangesComment ActionsChange 909796 merged by Dzahn:
[operations/puppet@production] gerrit: add host-based Hiera keys for gerrit1003
Comment ActionsChange 910049 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] site: add gerrit prod role to gerrit1003
Comment ActionsChange 910064 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add gerrit1003 to rsync dest hosts when using prod role
Comment ActionsChange 910064 merged by Dzahn:
[operations/puppet@production] gerrit: add gerrit1003 to rsync dest hosts when using prod role
Comment ActionsChange 909791 merged by Dzahn:
[operations/puppet@production] replace gerrit1001 with gerrit1003 as ping target for blackbox smoke
Comment ActionsChange 909790 merged by Dzahn:
[operations/puppet@production] acme_chief/gerrit certs: add gerrit1003 to hosts and gerrit-new to SNI
Comment ActionsChange 909795 merged by Dzahn:
[operations/puppet@production] cloudgw: allow VMs to speak to new gerrit server (gerrit1003)
Comment ActionsChange 910049 merged by Dzahn:
[operations/puppet@production] site: add gerrit prod role to gerrit1003
Comment ActionsMentioned in SAL (#wikimedia-operations) [2023-04-25T21:19:20Z] <mutante> gerrit1003 - chown -R gerrit2:gerrit2 /srv/gerrit T333143 T326368
Comment ActionsChange 911941 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: make configurable whether service is running
Comment ActionsChange 911941 abandoned by Dzahn:
[operations/puppet@production] gerrit: make configurable whether service is running
Reason:
not needed
Comment ActionsChange 914021 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add gerrit1003 to gerrit ssh_allowed hosts
Comment ActionsChange 914021 merged by Dzahn:
[operations/puppet@production] gerrit: add gerrit1003 to gerrit ssh_allowed hosts
hashar added projects: Release-Engineering-Team, Gerrit.May 5 2023, 3:28 PM2023-05-05 15:28:19 (UTC+0)Comment ActionsI wasn't aware about this task until yesterday (via T335730).
I'd like the new host to be added first as a replica rather than an entirely new primary Gerrit server then do a switch over of the service from the current primary gerrit1001 to the new gerrit1003.
Notably, I would like to avoid carrying state from a host to another since last time it caused multiple issues (notably we had obsolete left over files that never got garbage collected by Puppet, the caches filed the disk after a decade of being filing up). In theory the replica can be used as a primary as is, albeit with cold caches at first but I don't think it will cause problems in practice. Or we can do that for the gerrit2002 replacement?
Comment ActionsChange 916639 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/dns@master] gerrit: switch service name, turn new into current and current into old
Comment ActionsChange 916637 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/dns@master] lower TTL for gerrit.wikimedia.org reverse lookups
Comment Actions
In T326368#8829791, @hashar wrote:
I'd like the new host to be added first as a replica rather than an entirely new primary Gerrit server then do a switch over of the service from the current primary gerrit1001 to the new gerrit1003.
Are you suggesting to ONLY let gerrit replicate and no rsync at all or first let Gerrit replicate and then still rsync afterwards regardlesss? If the first then it would be different from how we did it last time but if the latter then would'nt we copy those unwanted files anyways?
Comment ActionsChange 916637 merged by Dzahn:
[operations/dns@master] lower TTL for gerrit.wikimedia.org reverse lookups
Comment ActionsChange 916639 merged by Dzahn:
[operations/dns@master] gerrit: switch service IP, turn new into current and current into old
Comment ActionsUpstream released a security update of Gerrit yesterday (3.5.6) I thus upgraded gerrit1001 and gerrit2002 to the new version this morning shortly after the train T336339.
I did not do gerrit1003 since it was not in the dsh group of operations/puppet which prompted me to create a new gerrit dsh group fully managed by Puppet ( https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/918482 ). So now scap deploy runs on all three hosts.
I checked gerrit1003 which is publicly reachable via gerrit-new.wikimedia.org but can't ssh into it. Turns out it does not have any of the git repositories under /srv/gerrit/git and thus lack any projects and users. I don't think I can do the Gerrit upgrade without them, we would need to rsync in the git repositories from the primary gerrit1001 to gerrit1003 (they are in /srv/gerrit/git).
https://phabricator.wikimedia.org/P47782. See my comments there for details.
The main thing is:
- changed the way we deploy the DNS change to avoid the problem that you can't merge in Gerrit when Gerrit is down :)
- upped bwlimit to 10/20x faster after confirming we really have gigabit
And some details like how long we are planning to keep the old host around, a decom ticket etc.
Comment Actions
In T326368#8842150, @hashar wrote:
..but can't ssh into it. Turns out it does not have any of the git repositories under /srv/gerrit/git and thus lack any projects and users. I don't think I can do the Gerrit upgrade without them, we would need to rsync..
ACK! This is in progress. I am currently working on fixing this. We had already (slowly, over 4 days originally) synced data and things worked but then today data got deleted again during our migration window. I am currently syncing again to get it back in order.
To speed things up so we can deploy asap (and before our new window tomorrow) , I am syncing with bwlimit 100m right now. I will update you shortly.
Comment Actions@hashar @thcipriani status update:
I have freshly:
- rsynced /srv/gerrit
- rsynced /var/lib/gerrit2
- for the lfs path change: copied /srv/gerrit/plugins/lfs to /srv/gerrit/data/lfs on gerrit1003, did not "mv", just "cp" for now so right now in both locations
- ensured it's all owned gerrit2:gerrit2
- restarted gerrit service
Now I would like you to try deploying again.
The thing is.. currently I see no repos in gerrit-new web UI at all.. even though everything was synced and before it did show them?!
Comment ActionsChange 918529 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/dns@master] Revert "Revert "gerrit: switch service IP, turn new into current and current into old""
Comment ActionsChange 918589 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: enable replication from gerrit1003, disable from gerrit1001
Comment Actionssteps we have to add:
- move/copy lfs data which is in new location on new host (for a good reason) (https://gerrit.wikimedia.org/r/c/operations/puppet/+/911363)
- disable gerrit1001 and enable gerrit1003 as the source of replication to gerrit2002 and github: https://gerrit.wikimedia.org/r/c/operations/puppet/+/918589
Comment ActionsChange 918529 merged by Dzahn:
[operations/dns@master] Revert "Revert "gerrit: switch service IP, turn new into current and current into old""
Comment ActionsChange 918589 merged by Dzahn:
[operations/puppet@production] gerrit: enable replication from gerrit1003, disable from gerrit1001
Comment ActionsChange 919151 had a related patch set uploaded (by Ssingh; author: Ssingh):
[operations/homer/public@master] definition/static.net: add Gerrit switchover IPs
Comment ActionsChange 919151 merged by Ssingh:
[operations/homer/public@master] definition/static.net: add Gerrit switchover IPs
Comment ActionsChange 919226 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] acme_chief: add gerrit-old to list of allowed SANs from gerrit hosts
Comment ActionsChange 919226 merged by Dzahn:
[operations/puppet@production] acme_chief: add gerrit-old to list of allowed SANs from gerrit hosts
Comment ActionsChange 919244 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: disable monitoring for gerrit1001
Comment ActionsChange 919246 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add parameter service_ensure, set to stopped on gerrit1001
Comment ActionsService is implemeted on gerrit1003.
It is now the production server behind gerrit.wikimedia.org since today.
A few follow-ups will be part of decomin'g the old machine (T336427).
Comment ActionsChange 919246 abandoned by Dzahn:
[operations/puppet@production] gerrit: add parameter service_ensure, set to stopped on gerrit1001
Reason:
replaced in favor of https://gerrit.wikimedia.org/r/c/operations/puppet/+/919359
Comment ActionsChange 919359 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: allow masking the service and do so on gerrit1001
Comment ActionsChange 919402 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add gerrit1003 to hosts using KexAlgo ecdh-sha2-nistp521 for ssh
Comment ActionsChange 919405 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add gerrit1003 SSH host key known_hosts
Comment ActionsChange 919405 merged by Dzahn:
[operations/puppet@production] gerrit: add gerrit1003 SSH host key known_hosts
Comment ActionsChange 919359 merged by Dzahn:
[operations/puppet@production] gerrit: allow masking the service and do so on gerrit1001
Comment ActionsChange 919244 merged by Dzahn:
[operations/puppet@production] gerrit: disable monitoring for gerrit1001
Comment ActionsChange 919402 merged by Dzahn:
[operations/puppet@production] gerrit: add gerrit1003 to hosts using KexAlgo ecdh-sha2-nistp521 for ssh
Comment ActionsChange 924608 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit/bacula: adjust Gerrit file paths to be backed up
Comment ActionsChange 924608 merged by Dzahn:
[operations/puppet@production] gerrit/bacula: adjust Gerrit file paths to be backed up
Comment ActionsChange 927280 had a related patch set uploaded (by Dzahn; author: Dzahn):
[integration/config@master] update gerrit.wikimedia.org IP in dockerfiles/maven-java8/gerrit_ssh_host_key
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Unknown Object (Task) | |||||
| Resolved | Papaul | T326366 Q3:rack/setup/install gerrit1003 | |||
| Open | None | T291916 Tracking task for Bullseye migrations in production | |||
| Resolved | Dzahn | T327068 Bullseye upgrade for remaining Collab hosts | |||
| Resolved | Dzahn | T326368 gerrit1003 service implementation task | |||
| Resolved | Dzahn | T334524 add new and remove old Gerrit service IP from netbox and DNS | |||
| Resolved | Dzahn | T336427 decom gerrit1001 | |||
| Resolved | Jclark-ctr | T340077 decommission gerrit1001.wikimedia.org (dcops, netbox) | |||
| Resolved | BUG REPORT | hashar | T336524 IPv6 SSH to gerrit.wikimedia.org hangs (blackhole route?) |
- Mentioned In
- T387833: Gerrit failover process
T338470: Rename gerrit2 unix user to gerrit and assign a fixed uid
T336524: IPv6 SSH to gerrit.wikimedia.org hangs (blackhole route?)
T335730: gerrit-replica seems out of date breaking extension distributor
T333143: Move Gerrit data out of root partition
T257317: scap deploy --init on deployment server fails on first puppet run
T334524: add new and remove old Gerrit service IP from netbox and DNS
T327068: Bullseye upgrade for remaining Collab hosts
T334521: upgrade gerrit servers to bullseye - Mentioned Here
- T336427: decom gerrit1001
T336339: Upgrade Gerrit from 3.5.5 to 3.5.6
P47782 gerrit migration plan/steps
T335730: gerrit-replica seems out of date breaking extension distributor
T333143: Move Gerrit data out of root partition
T257317: scap deploy --init on deployment server fails on first puppet run
T334521: upgrade gerrit servers to bullseye
T326366: Q3:rack/setup/install gerrit1003
Event Timeline
Change 909796 merged by Dzahn:
[operations/puppet@production] gerrit: add host-based Hiera keys for gerrit1003
Change 910049 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] site: add gerrit prod role to gerrit1003
Change 910064 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add gerrit1003 to rsync dest hosts when using prod role
Change 910064 merged by Dzahn:
[operations/puppet@production] gerrit: add gerrit1003 to rsync dest hosts when using prod role
Change 909791 merged by Dzahn:
[operations/puppet@production] replace gerrit1001 with gerrit1003 as ping target for blackbox smoke
Change 909790 merged by Dzahn:
[operations/puppet@production] acme_chief/gerrit certs: add gerrit1003 to hosts and gerrit-new to SNI
Change 909795 merged by Dzahn:
[operations/puppet@production] cloudgw: allow VMs to speak to new gerrit server (gerrit1003)
Change 910049 merged by Dzahn:
[operations/puppet@production] site: add gerrit prod role to gerrit1003
Mentioned in SAL (#wikimedia-operations) [2023-04-25T21:19:20Z] <mutante> gerrit1003 - chown -R gerrit2:gerrit2 /srv/gerrit T333143 T326368
Change 911941 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: make configurable whether service is running
Change 911941 abandoned by Dzahn:
[operations/puppet@production] gerrit: make configurable whether service is running
Reason:
not needed
Change 914021 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add gerrit1003 to gerrit ssh_allowed hosts
Change 914021 merged by Dzahn:
[operations/puppet@production] gerrit: add gerrit1003 to gerrit ssh_allowed hosts
I wasn't aware about this task until yesterday (via T335730).
I'd like the new host to be added first as a replica rather than an entirely new primary Gerrit server then do a switch over of the service from the current primary gerrit1001 to the new gerrit1003.
Notably, I would like to avoid carrying state from a host to another since last time it caused multiple issues (notably we had obsolete left over files that never got garbage collected by Puppet, the caches filed the disk after a decade of being filing up). In theory the replica can be used as a primary as is, albeit with cold caches at first but I don't think it will cause problems in practice. Or we can do that for the gerrit2002 replacement?
Change 916639 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/dns@master] gerrit: switch service name, turn new into current and current into old
Change 916637 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/dns@master] lower TTL for gerrit.wikimedia.org reverse lookups
In T326368#8829791, @hashar wrote:I'd like the new host to be added first as a replica rather than an entirely new primary Gerrit server then do a switch over of the service from the current primary gerrit1001 to the new gerrit1003.
Are you suggesting to ONLY let gerrit replicate and no rsync at all or first let Gerrit replicate and then still rsync afterwards regardlesss? If the first then it would be different from how we did it last time but if the latter then would'nt we copy those unwanted files anyways?
Change 916637 merged by Dzahn:
[operations/dns@master] lower TTL for gerrit.wikimedia.org reverse lookups
Change 916639 merged by Dzahn:
[operations/dns@master] gerrit: switch service IP, turn new into current and current into old
Upstream released a security update of Gerrit yesterday (3.5.6) I thus upgraded gerrit1001 and gerrit2002 to the new version this morning shortly after the train T336339.
I did not do gerrit1003 since it was not in the dsh group of operations/puppet which prompted me to create a new gerrit dsh group fully managed by Puppet ( https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/918482 ). So now scap deploy runs on all three hosts.
I checked gerrit1003 which is publicly reachable via gerrit-new.wikimedia.org but can't ssh into it. Turns out it does not have any of the git repositories under /srv/gerrit/git and thus lack any projects and users. I don't think I can do the Gerrit upgrade without them, we would need to rsync in the git repositories from the primary gerrit1001 to gerrit1003 (they are in /srv/gerrit/git).
The main thing is:
- changed the way we deploy the DNS change to avoid the problem that you can't merge in Gerrit when Gerrit is down :)
- upped bwlimit to 10/20x faster after confirming we really have gigabit
And some details like how long we are planning to keep the old host around, a decom ticket etc.
In T326368#8842150, @hashar wrote:..but can't ssh into it. Turns out it does not have any of the git repositories under /srv/gerrit/git and thus lack any projects and users. I don't think I can do the Gerrit upgrade without them, we would need to rsync..
ACK! This is in progress. I am currently working on fixing this. We had already (slowly, over 4 days originally) synced data and things worked but then today data got deleted again during our migration window. I am currently syncing again to get it back in order.
To speed things up so we can deploy asap (and before our new window tomorrow) , I am syncing with bwlimit 100m right now. I will update you shortly.
@hashar @thcipriani status update:
I have freshly:
- rsynced /srv/gerrit
- rsynced /var/lib/gerrit2
- for the lfs path change: copied /srv/gerrit/plugins/lfs to /srv/gerrit/data/lfs on gerrit1003, did not "mv", just "cp" for now so right now in both locations
- ensured it's all owned gerrit2:gerrit2
- restarted gerrit service
Now I would like you to try deploying again.
The thing is.. currently I see no repos in gerrit-new web UI at all.. even though everything was synced and before it did show them?!
Change 918529 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/dns@master] Revert "Revert "gerrit: switch service IP, turn new into current and current into old""
Change 918589 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: enable replication from gerrit1003, disable from gerrit1001
steps we have to add:
- move/copy lfs data which is in new location on new host (for a good reason) (https://gerrit.wikimedia.org/r/c/operations/puppet/+/911363)
- disable gerrit1001 and enable gerrit1003 as the source of replication to gerrit2002 and github: https://gerrit.wikimedia.org/r/c/operations/puppet/+/918589
Change 918529 merged by Dzahn:
[operations/dns@master] Revert "Revert "gerrit: switch service IP, turn new into current and current into old""
Change 918589 merged by Dzahn:
[operations/puppet@production] gerrit: enable replication from gerrit1003, disable from gerrit1001
Change 919151 had a related patch set uploaded (by Ssingh; author: Ssingh):
[operations/homer/public@master] definition/static.net: add Gerrit switchover IPs
Change 919151 merged by Ssingh:
[operations/homer/public@master] definition/static.net: add Gerrit switchover IPs
Change 919226 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] acme_chief: add gerrit-old to list of allowed SANs from gerrit hosts
Change 919226 merged by Dzahn:
[operations/puppet@production] acme_chief: add gerrit-old to list of allowed SANs from gerrit hosts
Change 919244 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: disable monitoring for gerrit1001
Change 919246 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add parameter service_ensure, set to stopped on gerrit1001
Service is implemeted on gerrit1003.
It is now the production server behind gerrit.wikimedia.org since today.
A few follow-ups will be part of decomin'g the old machine (T336427).
Change 919246 abandoned by Dzahn:
[operations/puppet@production] gerrit: add parameter service_ensure, set to stopped on gerrit1001
Reason:
replaced in favor of https://gerrit.wikimedia.org/r/c/operations/puppet/+/919359
Change 919359 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: allow masking the service and do so on gerrit1001
Change 919402 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add gerrit1003 to hosts using KexAlgo ecdh-sha2-nistp521 for ssh
Change 919405 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add gerrit1003 SSH host key known_hosts
Change 919405 merged by Dzahn:
[operations/puppet@production] gerrit: add gerrit1003 SSH host key known_hosts
Change 919359 merged by Dzahn:
[operations/puppet@production] gerrit: allow masking the service and do so on gerrit1001
Change 919244 merged by Dzahn:
[operations/puppet@production] gerrit: disable monitoring for gerrit1001
Change 919402 merged by Dzahn:
[operations/puppet@production] gerrit: add gerrit1003 to hosts using KexAlgo ecdh-sha2-nistp521 for ssh
Change 924608 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit/bacula: adjust Gerrit file paths to be backed up
Change 924608 merged by Dzahn:
[operations/puppet@production] gerrit/bacula: adjust Gerrit file paths to be backed up
Change 927280 had a related patch set uploaded (by Dzahn; author: Dzahn):
[integration/config@master] update gerrit.wikimedia.org IP in dockerfiles/maven-java8/gerrit_ssh_host_key