yolokube/ansible
yolokube/ansible
3
1
Fork 0
Code Issues 3 Pull requests 2 Projects Wiki Activity

Longhorn Update v1.5.1 #39

New issue
Closed
opened 2023-07-17 08:34:17 +02:00 by aaron · 10 comments
aaron commented 2023-07-17 08:34:17 +02:00
Owner
Copy link
longhorn-manager time="2023-07-17T06:32:06Z" level=error msg="Upgrade failed: upgrade API version failed: cannot create CRDAPIVersionSetting: Internal error occurred: failed calling webhook \"validator.longhorn.io\": failed to call webhook: Post \"https://longhorn-admission-webhook.longhorn-system.svc:9443/v1/ │
│ webhook/validaton?timeout=10s\": context deadline exceeded"                                                                                                                                                                                                                                                             │
│ longhorn-manager time="2023-07-17T06:32:06Z" level=info msg="Upgrade leader lost: worker3.yolokube.de"                                                                                                                                                                                                                  │
│ longhorn-manager time="2023-07-17T06:32:06Z" level=fatal msg="Error starting manager: upgrade API version failed: cannot create CRDAPIVersionSetting: Internal error occurred: failed calling webhook \"validator.longhorn.io\": failed to call webhook: Post \"https://longhorn-admission-webhook.longhorn-system.svc: │
│ 9443/v1/webhook/validaton?timeout=10s\": context deadline exceeded"

Most likely network/firewall issues.

``` longhorn-manager time="2023-07-17T06:32:06Z" level=error msg="Upgrade failed: upgrade API version failed: cannot create CRDAPIVersionSetting: Internal error occurred: failed calling webhook \"validator.longhorn.io\": failed to call webhook: Post \"https://longhorn-admission-webhook.longhorn-system.svc:9443/v1/ │ │ webhook/validaton?timeout=10s\": context deadline exceeded" │ │ longhorn-manager time="2023-07-17T06:32:06Z" level=info msg="Upgrade leader lost: worker3.yolokube.de" │ │ longhorn-manager time="2023-07-17T06:32:06Z" level=fatal msg="Error starting manager: upgrade API version failed: cannot create CRDAPIVersionSetting: Internal error occurred: failed calling webhook \"validator.longhorn.io\": failed to call webhook: Post \"https://longhorn-admission-webhook.longhorn-system.svc: │ │ 9443/v1/webhook/validaton?timeout=10s\": context deadline exceeded" ``` Most likely network/firewall issues.
aaron commented 2023-07-17 08:53:16 +02:00
Author
Owner
Copy link

when I disable the firewall and redeploy longhorn it does not work. However if I reenable the firewall it works.

when I disable the firewall and redeploy longhorn it does not work. However if I reenable the firewall it works.
aaron commented 2023-07-17 08:53:47 +02:00
Author
Owner
Copy link

however the webhook is still unreachable and changes can not be made

however the webhook is still unreachable and changes can not be made
tom commented 2023-07-27 04:32:17 +02:00
Owner
Copy link

Should hopefully be fixed with commit c41e025c88

Ongoing review...

Should hopefully be fixed with commit [c41e025c88](https://git.ar21.de/yolokube/ansible/commit/c41e025c88dda7ba74524fe20465c34f4d309693) Ongoing review...
tom self-assigned this 2023-07-27 04:34:00 +02:00
aaron commented 2023-07-28 00:06:50 +02:00
Author
Owner
Copy link

Longhorn is not syncing the first time it is deployed with argocd. It hangs at "Waiting for completion of hook batch/Job/longhorn-pre-upgrade".

If I deploy it via Helm with the same values it works.

It might be related to this issue: argoproj/argo-cd#6880

Longhorn is not syncing the first time it is deployed with argocd. It hangs at "Waiting for completion of hook batch/Job/longhorn-pre-upgrade". If I deploy it via Helm with the same values it works. It might be related to this issue: [argoproj/argo-cd#6880](https://github.com/argoproj/argo-cd/issues/6880)
aaron commented 2023-07-28 00:08:15 +02:00
Author
Owner
Copy link

Should hopefully be fixed with commit c41e025c88

Ongoing review...

The network issues are fixed successfully.

> Should hopefully be fixed with commit [c41e025c88](https://git.ar21.de/yolokube/ansible/commit/c41e025c88dda7ba74524fe20465c34f4d309693) > > Ongoing review... The network issues are fixed successfully.
🎉 1
tom commented 2023-08-03 07:04:21 +02:00
Owner
Copy link

Longhorn is not syncing the first time it is deployed with argocd. It hangs at "Waiting for completion of hook batch/Job/longhorn-pre-upgrade".

If I deploy it via Helm with the same values it works.

It might be related to this issue: argoproj/argo-cd#6880

Very strange... the sync process times out. If I manually deploy Longhorn via Helm and restart the sync process, the sync completes successfully. 👻

> Longhorn is not syncing the first time it is deployed with argocd. It hangs at "Waiting for completion of hook batch/Job/longhorn-pre-upgrade". > > If I deploy it via Helm with the same values it works. > > It might be related to this issue: [argoproj/argo-cd#6880](https://github.com/argoproj/argo-cd/issues/6880) Very strange... the sync process times out. If I manually deploy Longhorn via Helm and restart the sync process, the sync completes successfully. 👻
tom commented 2023-08-03 08:16:56 +02:00
Owner
Copy link

Oh man... Cilium still has the hairpin problem 🙄😬

Oh man... Cilium still has the hairpin problem 🙄😬
tom commented 2023-08-12 08:33:48 +02:00
Owner
Copy link

Checked versions:

  • success:

    • k8s: 1.26.4
    • argocd: 2.7.0
    • longhorn: 1.4.2

  • failed:

    • k8s: 1.26.4
    • argocd: 2.7.8
    • longhorn: 1.4.2 & 1.5.1
Checked versions: - success: - k8s: `1.26.4` - argocd: `2.7.0` - longhorn: `1.4.2` - failed: - k8s: `1.26.4` - argocd: `2.7.8` - longhorn: `1.4.2` & `1.5.1`
tom commented 2023-08-12 10:00:56 +02:00
Owner
Copy link

Current tested & recommended version combination:

  • k8s: latest 1.27.4
  • argocd: latest 2.8.0
  • longhorn: 1.4.2 live migration to 1.4.3, since 1.5.0 it's syncing until timeout 👈 @aaron
Current tested & recommended version combination: - k8s: latest `1.27.4` - argocd: latest `2.8.0` - longhorn: `1.4.2` live migration to `1.4.3`, since `1.5.0` it's syncing until timeout 👈 @aaron
👍 1
aaron added this to the Prod ready project 2023-08-29 17:13:29 +02:00
aaron modified the project from Prod ready to Updates 2023-08-30 01:05:54 +02:00
aaron changed title from Longhorn failing to start to Longhorn Update v1.5.1 2023-08-30 01:07:06 +02:00
tom commented 2023-11-25 15:50:09 +01:00
Owner
Copy link

Longhorn fixed the problem 🎉
https://github.com/longhorn/longhorn/issues/6623

Longhorn fixed the problem 🎉 https://github.com/longhorn/longhorn/issues/6623
❤️ 1
tom closed this issue 2023-11-25 15:50:09 +01:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
tn_upgrade_debian_to_13
ar-vault
ar-test
dev
No results found.
Labels
Clear labels   
system
CNI

  
system
Common

  
system
Core Deployments

  
system
Infrastructure

  
system
Storage

  
type
bug
Something is not working

  
type
enhancement
New feature

  
type
question
More information is needed

  
type
update
Software needs update

  
type
dependency-dashboard
Only for Renovate Dependency Dashboard

No labels
system
CNI
system
Common
system
Core Deployments
system
Infrastructure
system
Storage
type
bug
type
enhancement
type
question
type
update
type
dependency-dashboard
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Updates
Assignees
Clear assignees
No assignees
aaron
tom
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
yolokube/ansible#39
No description provided.